Oct 30 12:33:15.599300 ip-10-0-129-133 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 12:33:15.599311 ip-10-0-129-133 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 12:33:15.599317 ip-10-0-129-133 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 12:33:15.599599 ip-10-0-129-133 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 12:33:25.600678 ip-10-0-129-133 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 12:33:25.600696 ip-10-0-129-133 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e170ef7bbfbb4c4f9ddc828084546ec3 --
Oct 30 12:35:15.225647 ip-10-0-129-133 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:35:15.681212 ip-10-0-129-133 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:35:15.681212 ip-10-0-129-133 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:35:15.681212 ip-10-0-129-133 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:35:15.681212 ip-10-0-129-133 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:35:15.684685 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684617    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:15.684685 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684684    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:15.684685 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684688    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684692    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684696    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684699    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684708    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684711    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684714    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684717    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684720    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684723    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684725    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684728    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684730    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684733    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684735    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684738    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684741    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684743    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684746    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684748    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:15.684778 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684751    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684754    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684756    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684759    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684761    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684764    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684766    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684769    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684771    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684774    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684776    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684779    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684781    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684784    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684787    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684790    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684793    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684796    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684799    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684801    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:15.685235 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684804    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684807    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684809    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684811    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684814    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684817    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684820    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684822    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684824    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684827    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684829    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684832    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684834    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684837    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684840    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684843    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684846    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684849    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684851    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:15.685715 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684853    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684857    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684860    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684862    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684864    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684867    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684869    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684872    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684875    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684878    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684881    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684884    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684887    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684889    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684892    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684895    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684897    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684900    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684902    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:15.686161 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684905    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684909    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684913    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684916    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684918    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684921    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684923    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684926    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684929    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684932    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684934    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684937    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684940    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684942    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684945    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684947    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684949    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684952    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684954    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684957    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:15.686656 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684959    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684962    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684964    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684967    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684972    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.684976    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685736    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685743    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685746    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685749    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685751    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685754    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685757    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685759    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685762    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685764    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685767    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685770    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:15.687136 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685773    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685775    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685779    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685781    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685784    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685787    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685789    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685792    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685795    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685797    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685800    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685802    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685805    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685808    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685810    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685812    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685815    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685818    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685820    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685823    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:15.687595 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685826    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685828    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685833    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685837    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685840    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685842    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685846    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685849    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685851    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685854    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685857    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685859    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685862    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685864    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685868    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685871    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685873    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685876    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685878    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685882    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:15.688066 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685884    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685887    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685889    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685892    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685895    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685897    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685899    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685902    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685904    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685907    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685909    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685912    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685914    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685918    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685920    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685923    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685926    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685928    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685931    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:15.688567 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685936    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685940    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685943    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685945    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685948    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685951    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685953    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685956    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685958    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685961    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685963    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685966    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685969    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685971    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685974    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685976    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685979    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685981    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685984    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685987    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:15.689024 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685989    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685992    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685994    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685997    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.685999    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686002    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686005    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686008    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686010    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686013    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686015    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686018    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686021    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686025    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686027    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686566    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686573    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686577    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686580    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:15.689555 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686583    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686586    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686588    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686591    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686594    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686597    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686600    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686602    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686605    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686608    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686610    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686613    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686616    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686618    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686621    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686623    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686627    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686629    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686632    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686635    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:15.690011 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686638    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686641    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686643    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686646    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686649    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686651    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686654    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686656    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686659    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686662    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686664    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686667    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686670    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686672    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686675    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686678    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686680    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686683    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686685    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:15.690563 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686690    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686693    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686696    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686698    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686702    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686706    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686709    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686712    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686714    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686717    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686720    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686722    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686725    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686727    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686730    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686733    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686735    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686738    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686740    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:15.691025 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686743    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686746    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686748    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686751    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686753    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686756    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686758    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686762    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686764    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686767    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686769    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686772    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686774    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686778    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686781    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686783    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686786    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686788    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686792    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686796    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:15.691493 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686800    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686803    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686808    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686811    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686813    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686816    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686818    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686821    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686824    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686827    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686830    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686832    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686835    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686837    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686840    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686842    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686846    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686849    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686851    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686854    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:15.691966 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686856    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686859    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686861    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.686864    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.686961    2573 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.686971    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.686979    2573 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.686986    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.686992    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.686996    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687000    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687004    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687007    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687010    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687015    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687019    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687022    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687025    2573 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687028    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687031    2573 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687034    2573 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687038    2573 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687043    2573 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:35:15.692456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687046    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687049    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687053    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687056    2573 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687060    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687063    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687066    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687070    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687073    2573 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687076    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687079    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687083    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687086    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687090    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687095    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687098    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687101    2573 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687104    2573 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687107    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687111    2573 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687114    2573 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687117    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687120    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687125    2573 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:35:15.693005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687130    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687133    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687136    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687139    2573 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687143    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687146    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687149    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687153    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687156    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687159    2573 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687162    2573 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687166    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687169    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687172    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687175    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687179    2573 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687182    2573 flags.go:64] FLAG: --help="false"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687184    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-129-133.ec2.internal"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687188    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687191    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687194    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687197    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687202    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687205    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:35:15.693900 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687207    2573 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687211    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687213    2573 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687216    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687220    2573 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687222    2573 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687225    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687229    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687233    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687235    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687238    2573 flags.go:64] FLAG: --lock-file=""
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687241    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687244    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687247    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687253    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687256    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687259    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687262    2573 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687266    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687269    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687272    2573 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687275    2573 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687279    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687282    2573 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687286    2573 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:35:15.694918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687289    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687292    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687295    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687298    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687301    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687305    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687308    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687319    2573 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687322    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687325    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687328    2573 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687331    2573 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687334    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687338    2573 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687342    2573 flags.go:64] FLAG: --port="10250"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687345    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687362    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-034db7daabe8ccabb"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687366    2573 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687370    2573 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687373    2573 flags.go:64] FLAG: --register-node="true"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687376    2573 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687380    2573 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687383    2573 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687386    2573 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687389    2573 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687393    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:35:15.696027 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687397    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687400    2573 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687403    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687406    2573 flags.go:64] FLAG: --runonce="false"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687409    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687412    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687415    2573 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687418    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687421    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687425    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687428    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687432    2573 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687435    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687438    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687441    2573 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687444    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687448    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687451    2573 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687454    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687459    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687463    2573 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687466    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687471    2573 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687473    2573 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687476    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:35:15.697263 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687479    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687482    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687485    2573 flags.go:64] FLAG: --v="2"
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687490    2573 flags.go:64] FLAG: --version="false"
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687499    2573 flags.go:64] FLAG: --vmodule=""
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687504    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687508    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687644    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687648    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687651    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687653    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687656    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687659    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687661    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687664    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687667    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687670    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687672    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687676    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687679    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687682    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:15.698162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687685    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687687    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687690    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687693    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687695    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687698    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687702    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687705    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687707    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687710    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687713    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687715    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687718    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687721    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687724    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687728    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687731    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687735    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687738    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:15.698795 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687741    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687744    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687747    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687750    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687752    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687755    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687758    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687760    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687763    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687765    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687769    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687772    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687775    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687777    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687780    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687782    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687785    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687788    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687790    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687795    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:15.699584 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687797    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687800    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687802    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687805    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687808    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687810    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687813    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687816    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687818    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687821    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687824    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687826    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687829    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687832    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687835    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687838    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687840    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687843    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687846    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:15.700239 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687849    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687851    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687854    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687857    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687860    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687863    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687865    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687868    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687870    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687873    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687875    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687878    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687882    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687885    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687887    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687890    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687893    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687895    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687898    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687901    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:15.700730 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687903    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687906    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687909    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687911    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687914    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687917    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687920    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687922    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687926    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687930    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687933    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687936    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687938    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.687941    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.687968    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688004    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:15.701388 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688008    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688011    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688015    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688018    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688022    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688026    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688029    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688031    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688034    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688038    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688041    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688044    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688047    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688049    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688052    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688055    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688057    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688060    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688063    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:15.701776 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688065    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688068    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688071    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688074    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688077    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688080    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688083    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688086    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688088    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688091    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688094    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688096    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688099    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688102    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688105    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688107    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688110    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688112    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688115    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:15.702226 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688117    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688121    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688125    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688129    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688131    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688148    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688152    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688155    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688159    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688162    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688165    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688168    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688170    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688173    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688176    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688179    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688181    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688184    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688187    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:15.702759 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688190    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688192    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688195    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688197    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688200    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688202    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688205    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688208    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688210    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688213    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688215    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688218    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688221    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688223    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688226    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688228    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688234    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688236    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688239    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:15.703210 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688242    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688244    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688246    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688249    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688252    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688254    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688257    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688260    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688262    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688265    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688267    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688270    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688273    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688276    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688278    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688281    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688283    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688286    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688288    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688291    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:15.703712 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688294    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688296    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688299    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688302    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688304    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688307    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688309    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688312    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.688314    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.695736    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.695755    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.695769    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.695777    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695862    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695873    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695879    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695885    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695890    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695895    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695900    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:15.704177 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695904    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695909    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695914    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695918    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695922    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695927    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695931    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695936    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695940    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695944    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695949    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695954    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695958    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695962    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695966    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695971    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695975    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695979    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695983    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695988    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:15.704745 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695993    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.695997    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696003    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696008    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696012    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696017    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696021    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696025    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696029    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696033    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696037    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696041    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696045    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696050    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696055    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696059    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696063    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696067    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696072    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696077    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:15.705225 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696081    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696085    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696090    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696094    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696098    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696103    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696107    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696111    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696115    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696120    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696124    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696128    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696133    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696137    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696142    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696147    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696152    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696156    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696160    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:15.706100 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696164    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696169    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696173    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696178    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696182    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696186    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696191    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696196    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696200    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696205    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696210    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696214    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696218    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696223    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696227    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696231    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696235    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696240    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696244    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:15.706619 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696251    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696258    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696264    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696269    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696274    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696279    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696283    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696287    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696292    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696296    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696302    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696306    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696310    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696317    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696322    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696327    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696332    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696336    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696340    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696345    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:15.707185 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696365    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.696407    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696636    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696645    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696650    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696655    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696661    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696665    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696670    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696674    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696678    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696682    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696686    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696690    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696695    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696699    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:15.707849 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696704    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696708    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696712    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696716    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696720    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696725    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696730    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696735    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696739    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696744    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696748    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696752    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696757    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696761    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696765    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696769    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696775    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696779    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696783    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696787    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:15.708219 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696791    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696795    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696800    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696804    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696808    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696813    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696817    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696821    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696825    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696830    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696834    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696838    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696843    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696847    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696851    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696856    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696860    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696864    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696868    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:15.708722 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696873    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696879    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696884    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696888    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696893    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696897    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696901    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696905    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696910    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696914    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696918    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696922    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696926    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696931    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696935    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696939    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696943    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696947    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696951    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:15.709162 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696956    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696960    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696964    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696968    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696973    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696977    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696981    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696986    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696990    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.696994    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697000    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697006    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697011    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697015    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697020    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697025    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697029    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697036    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697042    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:15.709668 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697047    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697052    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697057    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697062    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697066    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697070    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697074    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697079    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697083    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697087    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697091    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697096    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697100    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697104    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697108    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.697145    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:35:15.710114 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697196    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697204    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697209    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697214    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697218    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697222    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697227    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697231    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697235    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697240    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697244    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697248    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697252    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697257    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697264    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697268    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697272    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697276    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697279    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:15.710517 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697283    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697287    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697292    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697296    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697301    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697305    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697310    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697314    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697318    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697323    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697327    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697331    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697335    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697339    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697345    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697366    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697372    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697376    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697380    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:15.710957 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697384    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697388    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697392    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697397    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697401    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697405    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697409    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697413    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697420    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697425    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697429    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697433    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697438    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697443    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697449    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697456    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697461    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697466    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697470    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:15.711418 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697476    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697481    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697485    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697491    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697499    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697505    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697511    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697517    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697523    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697527    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697532    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697537    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697542    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697546    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697550    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697555    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697559    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697563    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697567    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697572    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:15.711879 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697577    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697582    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697588    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697592    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697597    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697601    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697606    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697610    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697614    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697618    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697623    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697627    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697633    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697637    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697641    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697646    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697650    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697654    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697658    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697662    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:15.712344 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697667    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697671    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697677    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697681    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697686    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697690    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697694    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697698    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:15.697703    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.697780    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.701136    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.702557    2573 server.go:1014] "Starting client certificate rotation"
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.702650    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.703399    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.708225    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:35:15.712845 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.709373    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:35:15.723039 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.723021    2573 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:35:15.726743 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.726729    2573 log.go:25] "Validated CRI v1 image API"
Oct 30 12:35:15.727577 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.727553    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:35:15.734267 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.734244    2573 fs.go:135] Filesystem UUIDs: map[073ad6aa-9489-412b-be5d-817070a18bf6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 acda0c59-d135-4c6e-ac7f-e77170b4b540:/dev/nvme0n1p3]
Oct 30 12:35:15.734321 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.734268    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 12:35:15.739413 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.739298    2573 manager.go:217] Machine: {Timestamp:2026-10-30 12:35:15.737831439 +0000 UTC m=+0.382819568 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103594 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a5095fa5d468d0d88a55e83f5fc7c SystemUUID:ec2a5095-fa5d-468d-0d88-a55e83f5fc7c BootID:e170ef7b-bfbb-4c4f-9ddc-828084546ec3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c7:80:30:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c7:80:30:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:b3:1b:83:f5:35 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:35:15.739413 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.739412    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:35:15.739531 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.739480    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:35:15.742846 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.742824    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:35:15.742973 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.742848    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-133.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:35:15.743022 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.742975    2573 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:35:15.743022 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.742982    2573 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:35:15.743022 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.743013    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:35:15.743100 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.743025    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:35:15.744284 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.744275    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:35:15.749098 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.749084    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:35:15.749224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.749213    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:35:15.750078 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.750067    2573 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:35:15.750134 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.750084    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:35:15.750134 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.750097    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:35:15.750134 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.750112    2573 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:35:15.750134 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.750122    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:35:15.750451 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.750438    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:35:15.753523 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.753506    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:35:15.760282 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.760268    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:35:15.762213 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.762148    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:35:15.762299 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.762242    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:35:15.762475 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.762459    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:35:15.762593 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.762578    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:35:15.762651 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.762596    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:35:15.762651 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.762605    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:35:15.762651 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.762615    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:35:15.762651 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.762625    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:35:15.763512 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.763500    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:35:15.763547 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.763527    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:35:15.763547 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.763536    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:35:15.763547 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.763543    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:35:15.763618 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.763554    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:35:15.763618 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.763564    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:35:15.764208 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.764196    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:35:15.764238 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.764210    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:35:15.767809 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.767795    2573 server.go:1263] "Started kubelet"
Oct 30 12:35:15.767897 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.767870    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:35:15.767933 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.767889    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:35:15.767966 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.767957    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:35:15.768455 ip-10-0-129-133 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:35:15.769448 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.769430    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:35:15.771297 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.771284    2573 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:35:15.776675 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.776655    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:35:15.776811 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.776793    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:15.777397 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.777383    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:35:15.777538 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.777512    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:35:15.777612 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.777541    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:35:15.777612 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.777545    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:35:15.777738 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.777641    2573 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:35:15.777738 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.777648    2573 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:35:15.779197 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.779179    2573 factory.go:55] Registering systemd factory
Oct 30 12:35:15.779270 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.779199    2573 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:35:15.779335 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.779287    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 12:35:15.779431 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.779420    2573 factory.go:153] Registering CRI-O factory
Oct 30 12:35:15.779469 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.779435    2573 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:35:15.779497 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.779482    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:35:15.779525 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.779504    2573 factory.go:103] Registering Raw factory
Oct 30 12:35:15.779525 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.779517    2573 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:35:15.780565 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.780542    2573 manager.go:319] Starting recovery of all containers
Oct 30 12:35:15.787416 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.787389    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:35:15.789084 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.789064    2573 manager.go:324] Recovery completed
Oct 30 12:35:15.790951 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.790923    2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-133.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 12:35:15.791474 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.791404    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-133.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 12:35:15.793534 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.790142    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-133.ec2.internal.18e34ecc3f69b4c4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-133.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-133.ec2.internal,},FirstTimestamp:2026-10-30 12:35:15.767776452 +0000 UTC m=+0.412764582,LastTimestamp:2026-10-30 12:35:15.767776452 +0000 UTC m=+0.412764582,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-133.ec2.internal,}"
Oct 30 12:35:15.796430 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.796415    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:35:15.798157 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.798140    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:35:15.798232 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.798171    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:35:15.798232 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.798184    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:35:15.798690 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.798674    2573 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:35:15.798690 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.798687    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:35:15.798817 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.798704    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:35:15.798817 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.798714    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:35:15.799425 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.799410    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:35:15.799474 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.799437    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:35:15.799474 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.799447    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:35:15.800921 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.800908    2573 policy_none.go:50] "Start"
Oct 30 12:35:15.800968 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.800924    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:35:15.800968 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.800939    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:35:15.802914 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.802902    2573 policy_none.go:44] "Start"
Oct 30 12:35:15.809229 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.809207    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 12:35:15.809229 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.809209    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 12:35:15.809374 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.809322    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-133.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 12:35:15.827568 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.827490    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-133.ec2.internal.18e34ecc413943f8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-133.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-133.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-133.ec2.internal,},FirstTimestamp:2026-10-30 12:35:15.79815628 +0000 UTC m=+0.443144415,LastTimestamp:2026-10-30 12:35:15.79815628 +0000 UTC m=+0.443144415,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-133.ec2.internal,}"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.827941    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.830491    2573 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.830526    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.830537    2573 server.go:87] "Starting device plugin registration server"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.830753    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.830768    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.830984    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.831071    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.831079    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.831648    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.831692    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.844858    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.845948    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.845966    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.845980    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.846013    2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.849541    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-133.ec2.internal.18e34ecc413996c8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-133.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-133.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-133.ec2.internal,},FirstTimestamp:2026-10-30 12:35:15.79817748 +0000 UTC m=+0.443165614,LastTimestamp:2026-10-30 12:35:15.79817748 +0000 UTC m=+0.443165614,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-133.ec2.internal,}"
Oct 30 12:35:15.856235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.851271    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tx2mw"
Oct 30 12:35:15.860368 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.859628    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tx2mw"
Oct 30 12:35:15.874089 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.874021    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-133.ec2.internal.18e34ecc4139c722  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-133.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-133.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-133.ec2.internal,},FirstTimestamp:2026-10-30 12:35:15.798189858 +0000 UTC m=+0.443177989,LastTimestamp:2026-10-30 12:35:15.798189858 +0000 UTC m=+0.443177989,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-133.ec2.internal,}"
Oct 30 12:35:15.889049 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.889034    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:15.931675 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.931635    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:35:15.932402 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.932387    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:35:15.932464 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.932414    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:35:15.932464 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.932425    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:35:15.932464 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.932444    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-133.ec2.internal"
Oct 30 12:35:15.947000 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.946975    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"]
Oct 30 12:35:15.947066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.947054    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:35:15.949686 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.949673    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:35:15.949756 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.949697    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:35:15.949756 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.949710    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:35:15.950341 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.950329    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-133.ec2.internal"
Oct 30 12:35:15.950385 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.950347    2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-133.ec2.internal\": node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:15.951895 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.951883    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:35:15.952014 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.952003    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:15.952051 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.952027    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:35:15.953313 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.953300    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:35:15.953381 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.953324    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:35:15.953381 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.953337    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:35:15.953457 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.953416    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:35:15.953457 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.953438    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:35:15.953457 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.953447    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:35:15.955499 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.955484    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:15.955554 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.955509    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:35:15.956256 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.956241    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:35:15.956337 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.956270    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:35:15.956337 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.956283    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:35:15.978235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.978209    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ce8a1c7eb99ebcd1a86b5e56228777b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" (UID: \"7ce8a1c7eb99ebcd1a86b5e56228777b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:15.978235 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.978232    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ce8a1c7eb99ebcd1a86b5e56228777b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" (UID: \"7ce8a1c7eb99ebcd1a86b5e56228777b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:15.978389 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:15.978249    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3c78310b24bd820f4d32df7a2e462ac4-config\") pod \"kube-apiserver-proxy-ip-10-0-129-133.ec2.internal\" (UID: \"3c78310b24bd820f4d32df7a2e462ac4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:15.981068 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.981048    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-133.ec2.internal\" not found" node="ip-10-0-129-133.ec2.internal"
Oct 30 12:35:15.983472 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.983457    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-133.ec2.internal\" not found" node="ip-10-0-129-133.ec2.internal"
Oct 30 12:35:15.989806 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:15.989788    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:16.078340 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.078319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3c78310b24bd820f4d32df7a2e462ac4-config\") pod \"kube-apiserver-proxy-ip-10-0-129-133.ec2.internal\" (UID: \"3c78310b24bd820f4d32df7a2e462ac4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:16.078437 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.078342    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ce8a1c7eb99ebcd1a86b5e56228777b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" (UID: \"7ce8a1c7eb99ebcd1a86b5e56228777b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:16.078437 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.078381    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ce8a1c7eb99ebcd1a86b5e56228777b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" (UID: \"7ce8a1c7eb99ebcd1a86b5e56228777b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:16.078437 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.078407    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3c78310b24bd820f4d32df7a2e462ac4-config\") pod \"kube-apiserver-proxy-ip-10-0-129-133.ec2.internal\" (UID: \"3c78310b24bd820f4d32df7a2e462ac4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:16.078437 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.078424    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ce8a1c7eb99ebcd1a86b5e56228777b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" (UID: \"7ce8a1c7eb99ebcd1a86b5e56228777b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:16.078564 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.078424    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ce8a1c7eb99ebcd1a86b5e56228777b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" (UID: \"7ce8a1c7eb99ebcd1a86b5e56228777b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:16.090567 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:16.090546    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:16.190899 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:16.190847    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:16.286898 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.286875    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:16.289044 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.289030    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:16.292159 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:16.291830    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:16.392261 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:16.392230    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:16.492758 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:16.492709    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:16.593157 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:16.593135    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:16.693615 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:16.693594    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:16.701751 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.701734    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 12:35:16.701862 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.701845    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 12:35:16.794211 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:16.794181    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:16.859542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.859520    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:16.865567 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.865533    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:30:16 +0000 UTC" deadline="2028-06-01 10:43:09.705363147 +0000 UTC"
Oct 30 12:35:16.865567 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.865563    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13918h7m52.839803963s"
Oct 30 12:35:16.888581 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:16.888550    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce8a1c7eb99ebcd1a86b5e56228777b.slice/crio-66cf76fc27b029e8af3e440792091cf93c3a7ad4ca65b17c66d224468cc05d7c WatchSource:0}: Error finding container 66cf76fc27b029e8af3e440792091cf93c3a7ad4ca65b17c66d224468cc05d7c: Status 404 returned error can't find the container with id 66cf76fc27b029e8af3e440792091cf93c3a7ad4ca65b17c66d224468cc05d7c
Oct 30 12:35:16.891106 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:16.891081    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c78310b24bd820f4d32df7a2e462ac4.slice/crio-37962d5f41d40956e0372be06ffe4fa326336ed3132a67a554e4855e51592964 WatchSource:0}: Error finding container 37962d5f41d40956e0372be06ffe4fa326336ed3132a67a554e4855e51592964: Status 404 returned error can't find the container with id 37962d5f41d40956e0372be06ffe4fa326336ed3132a67a554e4855e51592964
Oct 30 12:35:16.894545 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:16.894521    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:16.896365 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.896349    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:35:16.923772 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.923738    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:16.978477 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.978448    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:16.998948 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:16.998928    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:35:17.000210 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.000196    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:17.018186 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.018130    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:35:17.150581 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.150559    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:17.528902 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.528878    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:17.751630 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.751594    2573 apiserver.go:52] "Watching apiserver"
Oct 30 12:35:17.775313 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.775279    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:35:17.775741 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.775715    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-9qbsn","kube-system/konnectivity-agent-j8d4p","kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal","openshift-cluster-node-tuning-operator/tuned-4sj5j","openshift-dns/node-resolver-5n22h","openshift-image-registry/node-ca-2s6fb","openshift-network-diagnostics/network-check-target-rqz8x","openshift-ovn-kubernetes/ovnkube-node-l7v4x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal","openshift-multus/network-metrics-daemon-crnn8","kube-system/global-pull-secret-syncer-9lfhr","openshift-multus/multus-9r4kp","openshift-multus/multus-additional-cni-plugins-6vxsv"]
Oct 30 12:35:17.778485 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.778456    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 12:35:17.778843 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.778825    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:17.778964 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:17.778912    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:17.781171 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.780868    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.783344 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.783315    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.785783 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.785760    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.788634 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.788556    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m442\" (UniqueName: \"kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442\") pod \"network-check-target-rqz8x\" (UID: \"b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3\") " pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:17.789695 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.789674    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:17.793405 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.793386    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:17.794536 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.794515    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.795631 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.795610    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-562t6\""
Oct 30 12:35:17.797194 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.797174    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:35:17.797310 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.797217    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:35:17.797310 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.797221    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:35:17.797310 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.797281    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:35:17.797310 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.797284    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:35:17.797496 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.797320    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:35:17.797856 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.797836    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:17.797965 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.797949    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:35:17.798142 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.798126    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:35:17.800651 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.800387    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:17.801928 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.801791    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:35:17.802028 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.801984    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:35:17.803028 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.803006    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.805522 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.805472    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:35:17.805632 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.805537    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:35:17.805632 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.805583    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:35:17.805753 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.805643    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:17.806493 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.806475    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:35:17.806609 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:17.806488    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:17.806749 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.806731    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:35:17.808453 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.808408    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:17.808558 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:17.808472    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:17.813005 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.812983    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:35:17.822117 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.822098    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:35:17.822972 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.822953    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xpmm7"
Oct 30 12:35:17.830486 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.830418    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xpmm7"
Oct 30 12:35:17.852606 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.852553    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal" event={"ID":"7ce8a1c7eb99ebcd1a86b5e56228777b","Type":"ContainerStarted","Data":"66cf76fc27b029e8af3e440792091cf93c3a7ad4ca65b17c66d224468cc05d7c"}
Oct 30 12:35:17.854121 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.854047    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal" event={"ID":"3c78310b24bd820f4d32df7a2e462ac4","Type":"ContainerStarted","Data":"37962d5f41d40956e0372be06ffe4fa326336ed3132a67a554e4855e51592964"}
Oct 30 12:35:17.879002 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.878975    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.888732    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-run-netns\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.888778    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.888797    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-socket-dir-parent\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.888816    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-cni-bin\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.888839    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-hostroot\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.888858    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prz95\" (UniqueName: \"kubernetes.io/projected/d22f52c3-6029-4b19-aa6a-9843e569dbc4-kube-api-access-prz95\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.888873    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-systemd\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.888889    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-registration-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.888915    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-cni-netd\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.888944    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.888969    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b382eac-dd89-486d-b80b-1f5a402f23d4-tmp-dir\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.888984    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-system-cni-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889008    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysconfig\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889031    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889058    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8s2g\" (UniqueName: \"kubernetes.io/projected/02a59e47-332d-40bb-af4a-2a0b59a816c8-kube-api-access-h8s2g\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.890090 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889075    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-etc-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889095    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovnkube-config\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889125    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889151    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6r5\" (UniqueName: \"kubernetes.io/projected/4b382eac-dd89-486d-b80b-1f5a402f23d4-kube-api-access-dd6r5\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889177    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-k8s-cni-cncf-io\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889199    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-daemon-config\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889223    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-var-lib-kubelet\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889248    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889276    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889304    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnz5h\" (UniqueName: \"kubernetes.io/projected/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-kube-api-access-bnz5h\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889326    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-cnibin\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889341    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-sys\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889357    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e69cda5-58f4-409b-8575-d0d8f3c09ba6-konnectivity-ca\") pod \"konnectivity-agent-j8d4p\" (UID: \"8e69cda5-58f4-409b-8575-d0d8f3c09ba6\") " pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889380    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvmw\" (UniqueName: \"kubernetes.io/projected/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-kube-api-access-cpvmw\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889402    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-kubernetes\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889445    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-env-overrides\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.890866 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889462    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889481    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-socket-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889499    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-kubelet\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889528    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-systemd\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-var-lib-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889564    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-cni-bin\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889590    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovnkube-script-lib\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889607    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnhcl\" (UniqueName: \"kubernetes.io/projected/a695456e-fedc-42fb-955b-5c55f356b0c3-kube-api-access-vnhcl\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889640    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m442\" (UniqueName: \"kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442\") pod \"network-check-target-rqz8x\" (UID: \"b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3\") " pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889662    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-iptables-alerter-script\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889697    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e69cda5-58f4-409b-8575-d0d8f3c09ba6-agent-certs\") pod \"konnectivity-agent-j8d4p\" (UID: \"8e69cda5-58f4-409b-8575-d0d8f3c09ba6\") " pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889754    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-slash\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889778    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-run-ovn-kubernetes\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889806    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889832    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-host\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889859    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsbfd\" (UniqueName: \"kubernetes.io/projected/7991e362-d106-4607-b8ee-ad70dfff9e7d-kube-api-access-jsbfd\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.891635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889885    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-host-slash\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889915    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-log-socket\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-cnibin\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889968    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.889994    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e14fcd05-8067-4312-b9bb-90c4c9a15b23-serviceca\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.890020    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-etc-kubernetes\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.890045    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-tuned\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891404    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-device-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891458    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-systemd-units\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-netns\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891523    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-modprobe-d\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891559    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4b382eac-dd89-486d-b80b-1f5a402f23d4-hosts-file\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891587    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c94f5d7-3c75-41bd-963f-a6fffa86747a-kubelet-config\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891614    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-cni-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891641    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d22f52c3-6029-4b19-aa6a-9843e569dbc4-cni-binary-copy\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-kubelet\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891695    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-multus-certs\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.892452 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891724    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-lib-modules\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891752    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysctl-conf\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891778    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7991e362-d106-4607-b8ee-ad70dfff9e7d-tmp\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891806    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vcf\" (UniqueName: \"kubernetes.io/projected/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-kube-api-access-d5vcf\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891839    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-ovn\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891864    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c94f5d7-3c75-41bd-963f-a6fffa86747a-dbus\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891896    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891922    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-node-log\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891948    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e14fcd05-8067-4312-b9bb-90c4c9a15b23-host\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.891980    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysctl-d\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.892007    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-run\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.892034    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovn-node-metrics-cert\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.892063    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-system-cni-dir\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.892093    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-cni-multus\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.892124    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg67k\" (UniqueName: \"kubernetes.io/projected/e14fcd05-8067-4312-b9bb-90c4c9a15b23-kube-api-access-kg67k\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.892152    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-conf-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.893193 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.892183    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-sys-fs\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.904612 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.904550    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:35:17.904612 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.904597    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:35:17.904612 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.904601    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gmdlh\""
Oct 30 12:35:17.904824 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.904773    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-scxms\""
Oct 30 12:35:17.904824 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.904821    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6qrlt\""
Oct 30 12:35:17.904932 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.904914    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2vwzh\""
Oct 30 12:35:17.904932 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.904920    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kkflg\""
Oct 30 12:35:17.905120 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.905038    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rkrpr\""
Oct 30 12:35:17.905120 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.905056    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x6m2k\""
Oct 30 12:35:17.905120 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.905093    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:35:17.905289 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.905198    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qjz2q\""
Oct 30 12:35:17.905289 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.905277    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:35:17.905389 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.905305    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:35:17.905389 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.905345    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:35:17.905389 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.905369    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:35:17.905582 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.905392    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:35:17.906805 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.906786    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:35:17.906922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.906845    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:35:17.906922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.906899    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:35:17.908135 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:17.908116    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:35:17.908135 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:17.908137    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:35:17.908279 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:17.908149    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2m442 for pod openshift-network-diagnostics/network-check-target-rqz8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:17.908279 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:17.908210    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442 podName:b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:18.408188817 +0000 UTC m=+3.053176956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2m442" (UniqueName: "kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442") pod "network-check-target-rqz8x" (UID: "b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:17.992768 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.992739    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-device-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.992918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.992778    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-systemd-units\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.992918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.992806    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-netns\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.992918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.992832    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-modprobe-d\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.992918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.992860    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4b382eac-dd89-486d-b80b-1f5a402f23d4-hosts-file\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:17.992918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.992886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c94f5d7-3c75-41bd-963f-a6fffa86747a-kubelet-config\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:17.992918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.992910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-cni-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.993226 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.992934    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d22f52c3-6029-4b19-aa6a-9843e569dbc4-cni-binary-copy\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.993226 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.992958    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-kubelet\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.993226 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.992984    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-multus-certs\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.993226 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993009    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-lib-modules\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.993226 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993033    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysctl-conf\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.993226 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993057    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7991e362-d106-4607-b8ee-ad70dfff9e7d-tmp\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.993226 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993082    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vcf\" (UniqueName: \"kubernetes.io/projected/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-kube-api-access-d5vcf\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:17.993226 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993108    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-ovn\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.993226 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993131    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c94f5d7-3c75-41bd-963f-a6fffa86747a-dbus\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:17.993226 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993155    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.993226 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993188    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-node-log\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.993226 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993211    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e14fcd05-8067-4312-b9bb-90c4c9a15b23-host\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993236    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysctl-d\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993267    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-run\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993293    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovn-node-metrics-cert\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993320    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-system-cni-dir\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993345    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-cni-multus\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993370    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg67k\" (UniqueName: \"kubernetes.io/projected/e14fcd05-8067-4312-b9bb-90c4c9a15b23-kube-api-access-kg67k\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993395    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-conf-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993435    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-sys-fs\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993466    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-run-netns\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993520    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-socket-dir-parent\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993546    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-cni-bin\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993570    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-hostroot\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prz95\" (UniqueName: \"kubernetes.io/projected/d22f52c3-6029-4b19-aa6a-9843e569dbc4-kube-api-access-prz95\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993619    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-systemd\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993645    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-registration-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.993812 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993674    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-cni-netd\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993702    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993730    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b382eac-dd89-486d-b80b-1f5a402f23d4-tmp-dir\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993755    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-system-cni-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993780    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysconfig\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993805    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993831    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8s2g\" (UniqueName: \"kubernetes.io/projected/02a59e47-332d-40bb-af4a-2a0b59a816c8-kube-api-access-h8s2g\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993859    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-etc-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovnkube-config\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993913    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993942    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6r5\" (UniqueName: \"kubernetes.io/projected/4b382eac-dd89-486d-b80b-1f5a402f23d4-kube-api-access-dd6r5\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993968    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-k8s-cni-cncf-io\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.993994    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-daemon-config\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994020    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-var-lib-kubelet\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994045    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994074    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994103    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnz5h\" (UniqueName: \"kubernetes.io/projected/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-kube-api-access-bnz5h\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:17.994542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-cnibin\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994164    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-sys\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994190    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e69cda5-58f4-409b-8575-d0d8f3c09ba6-konnectivity-ca\") pod \"konnectivity-agent-j8d4p\" (UID: \"8e69cda5-58f4-409b-8575-d0d8f3c09ba6\") " pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994217    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvmw\" (UniqueName: \"kubernetes.io/projected/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-kube-api-access-cpvmw\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994245    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-kubernetes\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994285    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-env-overrides\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994340    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-socket-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994383    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-kubelet\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994410    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-systemd\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994435    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-var-lib-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994466    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-cni-bin\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovnkube-script-lib\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994527    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhcl\" (UniqueName: \"kubernetes.io/projected/a695456e-fedc-42fb-955b-5c55f356b0c3-kube-api-access-vnhcl\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994572    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-iptables-alerter-script\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994606    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e69cda5-58f4-409b-8575-d0d8f3c09ba6-agent-certs\") pod \"konnectivity-agent-j8d4p\" (UID: \"8e69cda5-58f4-409b-8575-d0d8f3c09ba6\") " pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994632    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-slash\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.995224 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994658    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-run-ovn-kubernetes\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994688    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-host\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994742    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsbfd\" (UniqueName: \"kubernetes.io/projected/7991e362-d106-4607-b8ee-ad70dfff9e7d-kube-api-access-jsbfd\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994769    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-host-slash\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994797    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-log-socket\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994824    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-cnibin\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994852    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994878    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e14fcd05-8067-4312-b9bb-90c4c9a15b23-serviceca\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994906    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-etc-kubernetes\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.994932    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-tuned\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995329    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995383    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysconfig\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995445    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-device-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995486    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-systemd-units\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995522    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-netns\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995586    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995601    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-modprobe-d\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.996066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995653    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4b382eac-dd89-486d-b80b-1f5a402f23d4-hosts-file\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995729    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c94f5d7-3c75-41bd-963f-a6fffa86747a-kubelet-config\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995772    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-cni-multus\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995773    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-system-cni-dir\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995850    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-kubelet\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995896    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-multus-certs\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.995923    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-etc-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.996024    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-conf-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.996051    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-lib-modules\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.996085    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-sys-fs\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.996123    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-run-netns\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.996174    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysctl-conf\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:17.996196    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:17.996256    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret podName:8c94f5d7-3c75-41bd-963f-a6fffa86747a nodeName:}" failed. No retries permitted until 2026-10-30 12:35:18.496238821 +0000 UTC m=+3.141226965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret") pod "global-pull-secret-syncer-9lfhr" (UID: "8c94f5d7-3c75-41bd-963f-a6fffa86747a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.996523    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-socket-dir-parent\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.996561    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-cni-bin\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.996587    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-hostroot\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.997040 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.996619    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovnkube-config\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.996786    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-systemd\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.996828    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-registration-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.996858    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-cni-netd\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.996896    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-ovn\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997031    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c94f5d7-3c75-41bd-963f-a6fffa86747a-dbus\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997071    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-system-cni-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997089    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997060    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-cni-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997135    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997144    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-node-log\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997231    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysctl-d\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997260    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997285    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-run\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997459    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b382eac-dd89-486d-b80b-1f5a402f23d4-tmp-dir\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997524    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e14fcd05-8067-4312-b9bb-90c4c9a15b23-host\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997562    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-k8s-cni-cncf-io\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997582    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-var-lib-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.997922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997671    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d22f52c3-6029-4b19-aa6a-9843e569dbc4-cni-binary-copy\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.998753 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.997747    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-host\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.998753 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.998510    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-slash\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.998753 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.998581    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-run-ovn-kubernetes\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.998753 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.998618    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-cni-bin\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.998753 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.998695    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-cnibin\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:17.999004 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.998949    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-host-slash\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:17.999063 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.999002    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-log-socket\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.999063 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.999040    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-kubernetes\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.999176 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.999120    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-var-lib-kubelet\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:17.999229 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.999199    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-daemon-config\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:17.999229 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:17.999201    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:17.999332 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.999242    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.999332 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:17.999267    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs podName:4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd nodeName:}" failed. No retries permitted until 2026-10-30 12:35:18.499248546 +0000 UTC m=+3.144236681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs") pod "network-metrics-daemon-crnn8" (UID: "4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:17.999465 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.999325    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-kubelet\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.999465 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.999443    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-env-overrides\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.999572 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.999522    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-systemd\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:17.999572 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:17.999524    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-tuned\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:18.000492 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.000466    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-iptables-alerter-script\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:18.001740 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.001713    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:18.001740 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.001841    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:18.002198 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.002060    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovnkube-script-lib\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:18.002198 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.002191    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-socket-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:18.002880 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.002797    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-cnibin\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:18.003634 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.003010    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:18.003634 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.003174    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-sys\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:18.003634 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.003253    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-etc-kubernetes\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:18.003634 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.003272    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e69cda5-58f4-409b-8575-d0d8f3c09ba6-konnectivity-ca\") pod \"konnectivity-agent-j8d4p\" (UID: \"8e69cda5-58f4-409b-8575-d0d8f3c09ba6\") " pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:18.003634 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.003516    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e14fcd05-8067-4312-b9bb-90c4c9a15b23-serviceca\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:18.004764 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.004660    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovn-node-metrics-cert\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:18.007490 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.004932    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e69cda5-58f4-409b-8575-d0d8f3c09ba6-agent-certs\") pod \"konnectivity-agent-j8d4p\" (UID: \"8e69cda5-58f4-409b-8575-d0d8f3c09ba6\") " pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:18.007490 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.007123    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7991e362-d106-4607-b8ee-ad70dfff9e7d-tmp\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:18.014983 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.014960    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vcf\" (UniqueName: \"kubernetes.io/projected/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-kube-api-access-d5vcf\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:18.015248 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.015172    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg67k\" (UniqueName: \"kubernetes.io/projected/e14fcd05-8067-4312-b9bb-90c4c9a15b23-kube-api-access-kg67k\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:18.015565 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.015493    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8s2g\" (UniqueName: \"kubernetes.io/projected/02a59e47-332d-40bb-af4a-2a0b59a816c8-kube-api-access-h8s2g\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:18.016855 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.016814    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prz95\" (UniqueName: \"kubernetes.io/projected/d22f52c3-6029-4b19-aa6a-9843e569dbc4-kube-api-access-prz95\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:18.019917 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.019886    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvmw\" (UniqueName: \"kubernetes.io/projected/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-kube-api-access-cpvmw\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:18.019917 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.019901    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6r5\" (UniqueName: \"kubernetes.io/projected/4b382eac-dd89-486d-b80b-1f5a402f23d4-kube-api-access-dd6r5\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:18.020573 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.020529    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhcl\" (UniqueName: \"kubernetes.io/projected/a695456e-fedc-42fb-955b-5c55f356b0c3-kube-api-access-vnhcl\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:18.021115 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.021079    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnz5h\" (UniqueName: \"kubernetes.io/projected/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-kube-api-access-bnz5h\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:18.021801 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.021779    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsbfd\" (UniqueName: \"kubernetes.io/projected/7991e362-d106-4607-b8ee-ad70dfff9e7d-kube-api-access-jsbfd\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:18.097598 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.097533    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:18.110329 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:18.110301    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22f52c3_6029_4b19_aa6a_9843e569dbc4.slice/crio-e3245af77502ccbbbd098ab88a812a8132ac0eded9bd73dc334f4c1ae128688b WatchSource:0}: Error finding container e3245af77502ccbbbd098ab88a812a8132ac0eded9bd73dc334f4c1ae128688b: Status 404 returned error can't find the container with id e3245af77502ccbbbd098ab88a812a8132ac0eded9bd73dc334f4c1ae128688b
Oct 30 12:35:18.126209 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.126022    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:18.132375 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.132332    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:18.132375 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.132357    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:18.135893 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.135871    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:18.137685 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:18.137658    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7991e362_d106_4607_b8ee_ad70dfff9e7d.slice/crio-a65b267033d0234cef2f4b1bab51c8af329909023b54b61465909d2fa6955ddd WatchSource:0}: Error finding container a65b267033d0234cef2f4b1bab51c8af329909023b54b61465909d2fa6955ddd: Status 404 returned error can't find the container with id a65b267033d0234cef2f4b1bab51c8af329909023b54b61465909d2fa6955ddd
Oct 30 12:35:18.146493 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.146420    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:18.148764 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.148743    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:18.153585 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:18.153559    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b382eac_dd89_486d_b80b_1f5a402f23d4.slice/crio-d289cf88870519d59981a96755cec88250957564a6861f4391dedf8d3d97f2ab WatchSource:0}: Error finding container d289cf88870519d59981a96755cec88250957564a6861f4391dedf8d3d97f2ab: Status 404 returned error can't find the container with id d289cf88870519d59981a96755cec88250957564a6861f4391dedf8d3d97f2ab
Oct 30 12:35:18.158489 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.158469    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:18.160220 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:18.160090    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14fcd05_8067_4312_b9bb_90c4c9a15b23.slice/crio-19a7334ccbf0db5f2d47fe3f70e73ca8afa7540e29a1e9577d1f412e127c7055 WatchSource:0}: Error finding container 19a7334ccbf0db5f2d47fe3f70e73ca8afa7540e29a1e9577d1f412e127c7055: Status 404 returned error can't find the container with id 19a7334ccbf0db5f2d47fe3f70e73ca8afa7540e29a1e9577d1f412e127c7055
Oct 30 12:35:18.160405 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:18.160359    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a96919_4ab9_4bfe_9b8b_36d677c3d78c.slice/crio-8c29b7a0a904b247bd224e23897730fffa4f44bd1c25e2f70fb45888146c0773 WatchSource:0}: Error finding container 8c29b7a0a904b247bd224e23897730fffa4f44bd1c25e2f70fb45888146c0773: Status 404 returned error can't find the container with id 8c29b7a0a904b247bd224e23897730fffa4f44bd1c25e2f70fb45888146c0773
Oct 30 12:35:18.161447 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:18.161392    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda695456e_fedc_42fb_955b_5c55f356b0c3.slice/crio-bc4ec294735d21991682e0d19ea309829f1b4ad0b3248110acc14ccfe80a101b WatchSource:0}: Error finding container bc4ec294735d21991682e0d19ea309829f1b4ad0b3248110acc14ccfe80a101b: Status 404 returned error can't find the container with id bc4ec294735d21991682e0d19ea309829f1b4ad0b3248110acc14ccfe80a101b
Oct 30 12:35:18.165316 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:18.165289    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a59e47_332d_40bb_af4a_2a0b59a816c8.slice/crio-b36b5a2eed3f861a2d192395222ac197c11f9f212ec2b657ad28fcbc04709aa4 WatchSource:0}: Error finding container b36b5a2eed3f861a2d192395222ac197c11f9f212ec2b657ad28fcbc04709aa4: Status 404 returned error can't find the container with id b36b5a2eed3f861a2d192395222ac197c11f9f212ec2b657ad28fcbc04709aa4
Oct 30 12:35:18.170479 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:18.170441    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e69cda5_58f4_409b_8575_d0d8f3c09ba6.slice/crio-a4da9269af03503d70d162191d4d482a146238bd08e3b19d753f366476d3c5cf WatchSource:0}: Error finding container a4da9269af03503d70d162191d4d482a146238bd08e3b19d753f366476d3c5cf: Status 404 returned error can't find the container with id a4da9269af03503d70d162191d4d482a146238bd08e3b19d753f366476d3c5cf
Oct 30 12:35:18.181768 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.181747    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:18.193662 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:18.193635    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40905ce7_7c28_4d5c_a1a6_620f4d80dd5d.slice/crio-c504131be22c2c7ca2e34ce61ef4fb50a57b9edc5cb6e0064e554ba49424a67e WatchSource:0}: Error finding container c504131be22c2c7ca2e34ce61ef4fb50a57b9edc5cb6e0064e554ba49424a67e: Status 404 returned error can't find the container with id c504131be22c2c7ca2e34ce61ef4fb50a57b9edc5cb6e0064e554ba49424a67e
Oct 30 12:35:18.499619 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.499593    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:18.499724 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.499642    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m442\" (UniqueName: \"kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442\") pod \"network-check-target-rqz8x\" (UID: \"b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3\") " pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:18.499724 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.499697    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:18.499850 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:18.499736    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:18.499850 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:18.499799    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:18.499850 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:18.499807    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:35:18.499850 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:18.499827    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:35:18.499850 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:18.499848    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2m442 for pod openshift-network-diagnostics/network-check-target-rqz8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:18.500089 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:18.499815    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs podName:4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd nodeName:}" failed. No retries permitted until 2026-10-30 12:35:19.499799016 +0000 UTC m=+4.144787150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs") pod "network-metrics-daemon-crnn8" (UID: "4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:18.500089 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:18.499907    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442 podName:b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:19.499888887 +0000 UTC m=+4.144877013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2m442" (UniqueName: "kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442") pod "network-check-target-rqz8x" (UID: "b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:18.500089 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:18.499923    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret podName:8c94f5d7-3c75-41bd-963f-a6fffa86747a nodeName:}" failed. No retries permitted until 2026-10-30 12:35:19.499915978 +0000 UTC m=+4.144904149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret") pod "global-pull-secret-syncer-9lfhr" (UID: "8c94f5d7-3c75-41bd-963f-a6fffa86747a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:18.834658 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.834283    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:30:18 +0000 UTC" deadline="2028-06-02 09:25:17.272508588 +0000 UTC"
Oct 30 12:35:18.834658 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.834595    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13940h49m58.437921851s"
Oct 30 12:35:18.847481 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.846953    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:18.847481 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:18.847079    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:18.874457 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.874029    2573 generic.go:356] "Generic (PLEG): container finished" podID="7ce8a1c7eb99ebcd1a86b5e56228777b" containerID="28ea0fb2a0328000c0e582a37a2d0cc561aa84b070f00921efa2fa920c8e370b" exitCode=0
Oct 30 12:35:18.874457 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.874132    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal" event={"ID":"7ce8a1c7eb99ebcd1a86b5e56228777b","Type":"ContainerDied","Data":"28ea0fb2a0328000c0e582a37a2d0cc561aa84b070f00921efa2fa920c8e370b"}
Oct 30 12:35:18.883970 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.883936    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal" event={"ID":"3c78310b24bd820f4d32df7a2e462ac4","Type":"ContainerStarted","Data":"2d83ddc9065264a07479bb4e32edb16b0923d7e771df30d7b253d6f5c4f81a65"}
Oct 30 12:35:18.898886 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.898853    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x" event={"ID":"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d","Type":"ContainerStarted","Data":"c504131be22c2c7ca2e34ce61ef4fb50a57b9edc5cb6e0064e554ba49424a67e"}
Oct 30 12:35:18.910830 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.910793    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l" event={"ID":"02a59e47-332d-40bb-af4a-2a0b59a816c8","Type":"ContainerStarted","Data":"b36b5a2eed3f861a2d192395222ac197c11f9f212ec2b657ad28fcbc04709aa4"}
Oct 30 12:35:18.918820 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.918762    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vxsv" event={"ID":"a695456e-fedc-42fb-955b-5c55f356b0c3","Type":"ContainerStarted","Data":"bc4ec294735d21991682e0d19ea309829f1b4ad0b3248110acc14ccfe80a101b"}
Oct 30 12:35:18.921671 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.921622    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5n22h" event={"ID":"4b382eac-dd89-486d-b80b-1f5a402f23d4","Type":"ContainerStarted","Data":"d289cf88870519d59981a96755cec88250957564a6861f4391dedf8d3d97f2ab"}
Oct 30 12:35:18.925173 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.925053    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9r4kp" event={"ID":"d22f52c3-6029-4b19-aa6a-9843e569dbc4","Type":"ContainerStarted","Data":"e3245af77502ccbbbd098ab88a812a8132ac0eded9bd73dc334f4c1ae128688b"}
Oct 30 12:35:18.936536 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.936509    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j8d4p" event={"ID":"8e69cda5-58f4-409b-8575-d0d8f3c09ba6","Type":"ContainerStarted","Data":"a4da9269af03503d70d162191d4d482a146238bd08e3b19d753f366476d3c5cf"}
Oct 30 12:35:18.940953 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.940923    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9qbsn" event={"ID":"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c","Type":"ContainerStarted","Data":"8c29b7a0a904b247bd224e23897730fffa4f44bd1c25e2f70fb45888146c0773"}
Oct 30 12:35:18.942922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.942181    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal" podStartSLOduration=1.94216703 podStartE2EDuration="1.94216703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:18.941938232 +0000 UTC m=+3.586926377" watchObservedRunningTime="2026-10-30 12:35:18.94216703 +0000 UTC m=+3.587155174"
Oct 30 12:35:18.942922 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.942499    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2s6fb" event={"ID":"e14fcd05-8067-4312-b9bb-90c4c9a15b23","Type":"ContainerStarted","Data":"19a7334ccbf0db5f2d47fe3f70e73ca8afa7540e29a1e9577d1f412e127c7055"}
Oct 30 12:35:18.945592 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:18.945536    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4sj5j" event={"ID":"7991e362-d106-4607-b8ee-ad70dfff9e7d","Type":"ContainerStarted","Data":"a65b267033d0234cef2f4b1bab51c8af329909023b54b61465909d2fa6955ddd"}
Oct 30 12:35:19.509314 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:19.509232    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:19.509314 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:19.509284    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m442\" (UniqueName: \"kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442\") pod \"network-check-target-rqz8x\" (UID: \"b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3\") " pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:19.509569 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:19.509336    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:19.509569 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:19.509484    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:19.509569 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:19.509550    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret podName:8c94f5d7-3c75-41bd-963f-a6fffa86747a nodeName:}" failed. No retries permitted until 2026-10-30 12:35:21.509530526 +0000 UTC m=+6.154518648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret") pod "global-pull-secret-syncer-9lfhr" (UID: "8c94f5d7-3c75-41bd-963f-a6fffa86747a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:19.509741 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:19.509671    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:19.509741 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:19.509720    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs podName:4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd nodeName:}" failed. No retries permitted until 2026-10-30 12:35:21.509704211 +0000 UTC m=+6.154692331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs") pod "network-metrics-daemon-crnn8" (UID: "4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:19.509850 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:19.509806    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:35:19.509850 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:19.509827    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:35:19.509850 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:19.509840    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2m442 for pod openshift-network-diagnostics/network-check-target-rqz8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:19.509991 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:19.509881    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442 podName:b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:21.50986468 +0000 UTC m=+6.154852801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2m442" (UniqueName: "kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442") pod "network-check-target-rqz8x" (UID: "b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:19.835000 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:19.834875    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:30:18 +0000 UTC" deadline="2028-06-24 07:31:50.918605542 +0000 UTC"
Oct 30 12:35:19.835000 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:19.834911    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14466h56m31.083698785s"
Oct 30 12:35:19.849711 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:19.849675    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:19.849875 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:19.849833    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:19.850264 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:19.850238    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:19.850364 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:19.850345    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:19.958635 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:19.958591    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal" event={"ID":"7ce8a1c7eb99ebcd1a86b5e56228777b","Type":"ContainerStarted","Data":"a019a9fb7675a549ca3425934041846d4c4eb6d2f30a3ecf0690fee05f1f562e"}
Oct 30 12:35:19.994808 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:19.994743    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal" podStartSLOduration=2.994723761 podStartE2EDuration="2.994723761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:19.99412973 +0000 UTC m=+4.639117874" watchObservedRunningTime="2026-10-30 12:35:19.994723761 +0000 UTC m=+4.639711908"
Oct 30 12:35:20.846876 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:20.846622    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:20.846876 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:20.846768    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:20.986799 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:20.986725    2573 generic.go:356] "Generic (PLEG): container finished" podID="a695456e-fedc-42fb-955b-5c55f356b0c3" containerID="35cee61a413fa6eeb2d2484edb7f0e47749242e33a12eb3a12a7a90e53643eb0" exitCode=0
Oct 30 12:35:20.987930 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:20.987491    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vxsv" event={"ID":"a695456e-fedc-42fb-955b-5c55f356b0c3","Type":"ContainerDied","Data":"35cee61a413fa6eeb2d2484edb7f0e47749242e33a12eb3a12a7a90e53643eb0"}
Oct 30 12:35:21.530022 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:21.528806    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:21.530022 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:21.528892    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:21.530022 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:21.528939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m442\" (UniqueName: \"kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442\") pod \"network-check-target-rqz8x\" (UID: \"b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3\") " pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:21.530022 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:21.529107    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:35:21.530022 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:21.529126    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:35:21.530022 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:21.529142    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2m442 for pod openshift-network-diagnostics/network-check-target-rqz8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:21.530022 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:21.529208    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442 podName:b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:25.529188335 +0000 UTC m=+10.174176471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2m442" (UniqueName: "kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442") pod "network-check-target-rqz8x" (UID: "b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:21.530022 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:21.529799    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:21.530022 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:21.529866    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret podName:8c94f5d7-3c75-41bd-963f-a6fffa86747a nodeName:}" failed. No retries permitted until 2026-10-30 12:35:25.529848287 +0000 UTC m=+10.174836429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret") pod "global-pull-secret-syncer-9lfhr" (UID: "8c94f5d7-3c75-41bd-963f-a6fffa86747a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:21.530022 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:21.529940    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:21.530022 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:21.529980    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs podName:4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd nodeName:}" failed. No retries permitted until 2026-10-30 12:35:25.52996806 +0000 UTC m=+10.174956197 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs") pod "network-metrics-daemon-crnn8" (UID: "4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:21.850860 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:21.847594    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:21.850860 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:21.847741    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:21.850860 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:21.848387    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:21.850860 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:21.848491    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:22.846890 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:22.846807    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:22.847053 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:22.846947    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:22.994871 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:22.994833    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j8d4p" event={"ID":"8e69cda5-58f4-409b-8575-d0d8f3c09ba6","Type":"ContainerStarted","Data":"42ef771d4a3f03bd273ded4afbd2e398e4fc628e0e3dfc2446f2d17fd88b91e9"}
Oct 30 12:35:23.850393 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:23.849318    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:23.850393 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:23.849469    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:23.850631 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:23.850496    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:23.850631 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:23.850596    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:24.000437 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:24.000403    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2s6fb" event={"ID":"e14fcd05-8067-4312-b9bb-90c4c9a15b23","Type":"ContainerStarted","Data":"8fa1cd3c183029d481d52e109d3532e7137794054da995da3fe3049bbec520b7"}
Oct 30 12:35:24.032415 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:24.032347    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j8d4p" podStartSLOduration=3.8631911260000003 podStartE2EDuration="8.032329149s" totalImagesPullingTime="4.169138023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:23.026470961 +0000 UTC m=+7.671459101" watchObservedRunningTime="2026-10-30 12:35:24.032329149 +0000 UTC m=+8.677317291"
Oct 30 12:35:24.032649 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:24.032623    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2s6fb" podStartSLOduration=2.7144043350000002 podStartE2EDuration="8.03261587s" totalImagesPullingTime="5.318211535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:24.031943894 +0000 UTC m=+8.676932051" watchObservedRunningTime="2026-10-30 12:35:24.03261587 +0000 UTC m=+8.677604011"
Oct 30 12:35:24.846174 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:24.846115    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:24.846339 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:24.846214    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:25.004787 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:25.004725    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l" event={"ID":"02a59e47-332d-40bb-af4a-2a0b59a816c8","Type":"ContainerStarted","Data":"bdabce19d556fa9d9daedd287b32e0e13f192780fefd9ff4676a90f03267cc97"}
Oct 30 12:35:25.567151 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:25.567110    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:25.567320 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:25.567176    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m442\" (UniqueName: \"kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442\") pod \"network-check-target-rqz8x\" (UID: \"b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3\") " pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:25.567320 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:25.567228    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:25.567459 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:25.567368    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:25.567459 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:25.567430    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret podName:8c94f5d7-3c75-41bd-963f-a6fffa86747a nodeName:}" failed. No retries permitted until 2026-10-30 12:35:33.56741256 +0000 UTC m=+18.212400690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret") pod "global-pull-secret-syncer-9lfhr" (UID: "8c94f5d7-3c75-41bd-963f-a6fffa86747a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:25.567854 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:25.567835    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:25.567950 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:25.567887    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs podName:4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd nodeName:}" failed. No retries permitted until 2026-10-30 12:35:33.567871752 +0000 UTC m=+18.212859887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs") pod "network-metrics-daemon-crnn8" (UID: "4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:25.568025 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:25.567971    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:35:25.568025 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:25.567988    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:35:25.568025 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:25.567999    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2m442 for pod openshift-network-diagnostics/network-check-target-rqz8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:25.568167 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:25.568034    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442 podName:b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:33.568023406 +0000 UTC m=+18.213011539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2m442" (UniqueName: "kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442") pod "network-check-target-rqz8x" (UID: "b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:25.846516 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:25.846486    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:25.846655 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:25.846626    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:25.847029 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:25.847007    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:25.847125 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:25.847104    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:25.870259 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:25.870230    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:35:26.009220 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:26.009189    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l" event={"ID":"02a59e47-332d-40bb-af4a-2a0b59a816c8","Type":"ContainerStarted","Data":"31802fe21e28e36d40d2b5894919cf7db381ebd607e3cdcbf5743bcfb0883d0d"}
Oct 30 12:35:26.013580 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:26.013402    2573 generic.go:356] "Generic (PLEG): container finished" podID="a695456e-fedc-42fb-955b-5c55f356b0c3" containerID="21a96d86abdcd70e111a6f7d907478cd52280d12279720aef58a860303d8fa8e" exitCode=0
Oct 30 12:35:26.013580 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:26.013449    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vxsv" event={"ID":"a695456e-fedc-42fb-955b-5c55f356b0c3","Type":"ContainerDied","Data":"21a96d86abdcd70e111a6f7d907478cd52280d12279720aef58a860303d8fa8e"}
Oct 30 12:35:26.837066 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:26.836958    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:35:25.870253159Z","UUID":"43a944ad-acae-4d3f-82e4-408e9f6da32b","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:35:26.838988 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:26.838783    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:35:26.839096 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:26.838999    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:35:26.846748 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:26.846726    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:26.846881 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:26.846847    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:27.022715 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:27.022681    2573 generic.go:356] "Generic (PLEG): container finished" podID="a695456e-fedc-42fb-955b-5c55f356b0c3" containerID="256852000c5b7a52e7b880cb7158029c24bad61e29aa4269ce6988c33ec927a0" exitCode=0
Oct 30 12:35:27.023058 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:27.022785    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vxsv" event={"ID":"a695456e-fedc-42fb-955b-5c55f356b0c3","Type":"ContainerDied","Data":"256852000c5b7a52e7b880cb7158029c24bad61e29aa4269ce6988c33ec927a0"}
Oct 30 12:35:27.024989 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:27.024964    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5n22h" event={"ID":"4b382eac-dd89-486d-b80b-1f5a402f23d4","Type":"ContainerStarted","Data":"0e1d4c626d7bd4f6840df01749bdfc44557080bbb0b7daf03c4e96de01d41103"}
Oct 30 12:35:27.027988 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:27.027966    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9r4kp" event={"ID":"d22f52c3-6029-4b19-aa6a-9843e569dbc4","Type":"ContainerStarted","Data":"89223c7e9ce126ce24ca463dd96e69af261996192eb9956ac27e9e7e2248271a"}
Oct 30 12:35:27.120825 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:27.120767    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9r4kp" podStartSLOduration=3.265526207 podStartE2EDuration="11.120750422s" totalImagesPullingTime="7.855224215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:27.090785268 +0000 UTC m=+11.735773408" watchObservedRunningTime="2026-10-30 12:35:27.120750422 +0000 UTC m=+11.765738562"
Oct 30 12:35:27.121174 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:27.121142    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5n22h" podStartSLOduration=2.976827373 podStartE2EDuration="11.121130917s" totalImagesPullingTime="8.144303544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:27.120588687 +0000 UTC m=+11.765576808" watchObservedRunningTime="2026-10-30 12:35:27.121130917 +0000 UTC m=+11.766119057"
Oct 30 12:35:27.308131 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:27.308095    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:27.309010 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:27.308986    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:27.846542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:27.846507    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:27.846542 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:27.846528    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:27.846769 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:27.846654    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:27.847077 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:27.847057    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:28.032239 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:28.032209    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9qbsn" event={"ID":"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c","Type":"ContainerStarted","Data":"b7f66f5c0d8397a42d20dea2c801a46d4075355f55a54ba62dfac7b47ed8bf96"}
Oct 30 12:35:28.034744 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:28.034722    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l" event={"ID":"02a59e47-332d-40bb-af4a-2a0b59a816c8","Type":"ContainerStarted","Data":"f414bf54030beee4e7ccd64d08cd21b9960a9fa8a06ce15ae0ed883b1c1949d1"}
Oct 30 12:35:28.037243 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:28.037162    2573 generic.go:356] "Generic (PLEG): container finished" podID="a695456e-fedc-42fb-955b-5c55f356b0c3" containerID="9046885f3871eaa9712d2c06638f068ecdf50bc95c25f2575167d649150546b0" exitCode=0
Oct 30 12:35:28.037243 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:28.037194    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vxsv" event={"ID":"a695456e-fedc-42fb-955b-5c55f356b0c3","Type":"ContainerDied","Data":"9046885f3871eaa9712d2c06638f068ecdf50bc95c25f2575167d649150546b0"}
Oct 30 12:35:28.063088 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:28.063037    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9qbsn" podStartSLOduration=4.057539271 podStartE2EDuration="12.063022755s" totalImagesPullingTime="8.005483484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:28.062541517 +0000 UTC m=+12.707529658" watchObservedRunningTime="2026-10-30 12:35:28.063022755 +0000 UTC m=+12.708010893"
Oct 30 12:35:28.135941 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:28.135188    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l" podStartSLOduration=3.702479604 podStartE2EDuration="12.135169885s" totalImagesPullingTime="8.432690281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:28.134922127 +0000 UTC m=+12.779910267" watchObservedRunningTime="2026-10-30 12:35:28.135169885 +0000 UTC m=+12.780158025"
Oct 30 12:35:28.846563 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:28.846532    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:28.846718 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:28.846663    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:29.041142 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:29.041100    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4sj5j" event={"ID":"7991e362-d106-4607-b8ee-ad70dfff9e7d","Type":"ContainerStarted","Data":"3798fe87057f539184bfcdba44794b5257b55b0a4a8b5b0831064594a643f5c4"}
Oct 30 12:35:29.077297 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:29.077238    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4sj5j" podStartSLOduration=3.249067148 podStartE2EDuration="13.077220518s" totalImagesPullingTime="9.82815337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:29.076465692 +0000 UTC m=+13.721453835" watchObservedRunningTime="2026-10-30 12:35:29.077220518 +0000 UTC m=+13.722208663"
Oct 30 12:35:29.846928 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:29.846893    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:29.847093 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:29.847031    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:29.847093 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:29.847063    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:29.847235 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:29.847212    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:30.045706 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:30.045653    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vxsv" event={"ID":"a695456e-fedc-42fb-955b-5c55f356b0c3","Type":"ContainerStarted","Data":"de3ac4bae78734c1144d37c04108e4c319fd268acef6608ddf4520b91bb9af2b"}
Oct 30 12:35:30.846170 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:30.846144    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:30.846325 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:30.846253    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:31.051270 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:31.051234    2573 generic.go:356] "Generic (PLEG): container finished" podID="a695456e-fedc-42fb-955b-5c55f356b0c3" containerID="de3ac4bae78734c1144d37c04108e4c319fd268acef6608ddf4520b91bb9af2b" exitCode=0
Oct 30 12:35:31.051846 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:31.051287    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vxsv" event={"ID":"a695456e-fedc-42fb-955b-5c55f356b0c3","Type":"ContainerDied","Data":"de3ac4bae78734c1144d37c04108e4c319fd268acef6608ddf4520b91bb9af2b"}
Oct 30 12:35:31.846796 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:31.846550    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:31.846964 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:31.846562    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:31.846964 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:31.846916    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:31.847074 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:31.846982    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:32.058710 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:32.058668    2573 generic.go:356] "Generic (PLEG): container finished" podID="a695456e-fedc-42fb-955b-5c55f356b0c3" containerID="d4c4fde9d87a32db8f2978df239f925fd33b8f350ca08623e5fcbd4f0d29e6bc" exitCode=0
Oct 30 12:35:32.059126 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:32.058724    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vxsv" event={"ID":"a695456e-fedc-42fb-955b-5c55f356b0c3","Type":"ContainerDied","Data":"d4c4fde9d87a32db8f2978df239f925fd33b8f350ca08623e5fcbd4f0d29e6bc"}
Oct 30 12:35:32.846629 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:32.846611    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:32.846719 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:32.846692    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:33.064300 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:33.064078    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x" event={"ID":"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d","Type":"ContainerStarted","Data":"b12e3244e849d847faad5f919ed93ef316ebe0c4b089e1aa0889ef0ff9984879"}
Oct 30 12:35:33.064607 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:33.064375    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x" event={"ID":"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d","Type":"ContainerStarted","Data":"bcbed8244743612388eb1dd3b61cd763df8a339f9b82631cf4a71ab4e23de1da"}
Oct 30 12:35:33.064607 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:33.064417    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x" event={"ID":"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d","Type":"ContainerStarted","Data":"10da65371d19782653697addfff707abe675e30b370353f428f9c9287c798c22"}
Oct 30 12:35:33.064607 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:33.064453    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x" event={"ID":"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d","Type":"ContainerStarted","Data":"b0cc0e3c783b278172ed3315ddd6a212de8525d2fb763ae7773fd5f80acfd489"}
Oct 30 12:35:33.067702 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:33.067661    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vxsv" event={"ID":"a695456e-fedc-42fb-955b-5c55f356b0c3","Type":"ContainerStarted","Data":"9accd93149d39277a377a14f013c3b8a23fc1b066b2be7994ce30aa8bf17e6ea"}
Oct 30 12:35:33.110716 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:33.110681    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6vxsv" podStartSLOduration=6.935834784 podStartE2EDuration="17.110668325s" totalImagesPullingTime="10.174833541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:33.11064499 +0000 UTC m=+17.755633111" watchObservedRunningTime="2026-10-30 12:35:33.110668325 +0000 UTC m=+17.755656454"
Oct 30 12:35:33.631697 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:33.631669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:33.631847 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:33.631708    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:33.631847 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:33.631750    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m442\" (UniqueName: \"kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442\") pod \"network-check-target-rqz8x\" (UID: \"b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3\") " pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:33.631847 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:33.631823    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:33.631847 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:33.631825    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:33.631962 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:33.631856    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:35:33.631962 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:33.631871    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:35:33.631962 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:33.631882    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2m442 for pod openshift-network-diagnostics/network-check-target-rqz8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:33.631962 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:33.631886    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs podName:4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd nodeName:}" failed. No retries permitted until 2026-10-30 12:35:49.631866321 +0000 UTC m=+34.276854442 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs") pod "network-metrics-daemon-crnn8" (UID: "4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:35:33.631962 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:33.631904    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret podName:8c94f5d7-3c75-41bd-963f-a6fffa86747a nodeName:}" failed. No retries permitted until 2026-10-30 12:35:49.631895723 +0000 UTC m=+34.276883844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret") pod "global-pull-secret-syncer-9lfhr" (UID: "8c94f5d7-3c75-41bd-963f-a6fffa86747a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:35:33.631962 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:33.631919    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442 podName:b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:49.631911134 +0000 UTC m=+34.276899256 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2m442" (UniqueName: "kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442") pod "network-check-target-rqz8x" (UID: "b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:35:33.847250 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:33.847218    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:33.847250 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:33.847239    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:33.847517 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:33.847340    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:33.847517 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:33.847451    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:34.072316 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:34.072288    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x" event={"ID":"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d","Type":"ContainerStarted","Data":"5e8d5abbe6eef36ac3018f8fa12f0e15ff3b37817ab30865d770766643b92dd8"}
Oct 30 12:35:34.072688 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:34.072363    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x" event={"ID":"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d","Type":"ContainerStarted","Data":"db7fb1492ed4432f82c9e62530533e2f4c6b33a2fe71f4aba9c0bd6e04d00d8e"}
Oct 30 12:35:34.846647 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:34.846605    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:34.846796 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:34.846712    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:35.847920 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:35.847888    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:35.848378 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:35.847989    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:35.848378 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:35.848019    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:35.848378 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:35.848077    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:36.079689 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:36.079652    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x" event={"ID":"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d","Type":"ContainerStarted","Data":"46adf6cb2a279c0a9cc60cf8f300f07ce0731de9054a2a03c5f98c8ed3081026"}
Oct 30 12:35:36.846154 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:36.846125    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:36.846347 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:36.846218    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:37.314724 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:37.314697    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:37.315425 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:37.314829    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:35:37.315425 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:37.315419    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:37.846540 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:37.846511    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:37.846701 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:37.846672    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:37.846761 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:37.846714    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:37.846826 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:37.846791    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:38.086891 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:38.086858    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x" event={"ID":"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d","Type":"ContainerStarted","Data":"4faf4c9e26c6c82f86b62fd5966c2de294e228606c47ab7f37f7cefa2e6f2609"}
Oct 30 12:35:38.087166 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:38.087085    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:38.101821 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:38.101641    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:38.154816 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:38.154774    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x" podStartSLOduration=7.528393569 podStartE2EDuration="22.154762818s" totalImagesPullingTime="14.626369249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:38.144747197 +0000 UTC m=+22.789735371" watchObservedRunningTime="2026-10-30 12:35:38.154762818 +0000 UTC m=+22.799750986"
Oct 30 12:35:38.846703 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:38.846674    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:38.847114 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:38.846766    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:39.089757 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:39.089709    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:39.091714 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:39.090045    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:39.105560 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:39.105493    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:39.279373 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:39.279325    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-crnn8"]
Oct 30 12:35:39.279702 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:39.279674    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:39.279820 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:39.279798    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:39.280094 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:39.280072    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9lfhr"]
Oct 30 12:35:39.280218 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:39.280199    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:39.280328 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:39.280304    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:39.280748 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:39.280724    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rqz8x"]
Oct 30 12:35:39.280814 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:39.280808    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:39.280903 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:39.280880    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:40.847101 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:40.847076    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:40.847577 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:40.847102    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:40.847577 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:40.847172    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:40.847577 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:40.847191    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:40.847577 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:40.847261    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:40.847577 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:40.847324    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:42.847100 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:42.847072    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:42.847604 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:42.847173    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:42.847604 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:42.847252    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:42.847604 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:42.847377    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:42.847604 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:42.847420    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:42.847604 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:42.847493    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:44.846465 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:44.846432    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:44.846465 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:44.846454    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:44.846921 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:44.846570    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crnn8" podUID="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd"
Oct 30 12:35:44.846921 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:44.846614    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:44.846921 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:44.846727    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lfhr" podUID="8c94f5d7-3c75-41bd-963f-a6fffa86747a"
Oct 30 12:35:44.846921 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:44.846808    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqz8x" podUID="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3"
Oct 30 12:35:44.887541 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:44.887508    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:45.029967 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.029935    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:45.234862 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.234774    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:45.316846 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.316814    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:45.717867 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.717842    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeReady"
Oct 30 12:35:45.718017 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.717951    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 12:35:45.774522 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.774498    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pfqxj"]
Oct 30 12:35:45.826199 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.826173    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x5ck2"]
Oct 30 12:35:45.826373 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.826300    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pfqxj"
Oct 30 12:35:45.839146 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.839120    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pfqxj"]
Oct 30 12:35:45.839268 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.839162    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x5ck2"]
Oct 30 12:35:45.839268 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.839247    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:45.839504 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.839484    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 12:35:45.839564 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.839537    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 12:35:45.839619 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.839572    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 12:35:45.839619 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.839583    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pdckv\""
Oct 30 12:35:45.923822 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.923788    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57060355-8a18-4a06-93bb-af1de546c2b7-config-volume\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:45.924181 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.923844    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655h6\" (UniqueName: \"kubernetes.io/projected/57060355-8a18-4a06-93bb-af1de546c2b7-kube-api-access-655h6\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:45.924181 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.923915    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:45.924181 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.923971    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvh74\" (UniqueName: \"kubernetes.io/projected/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed-kube-api-access-rvh74\") pod \"ingress-canary-pfqxj\" (UID: \"bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed\") " pod="openshift-ingress-canary/ingress-canary-pfqxj"
Oct 30 12:35:45.924181 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.924007    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed-cert\") pod \"ingress-canary-pfqxj\" (UID: \"bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed\") " pod="openshift-ingress-canary/ingress-canary-pfqxj"
Oct 30 12:35:45.924181 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.924027    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57060355-8a18-4a06-93bb-af1de546c2b7-tmp-dir\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:45.939744 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.939718    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 12:35:45.939858 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.939792    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 12:35:45.939901 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:45.939885    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tz7hb\""
Oct 30 12:35:46.025321 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.025291    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvh74\" (UniqueName: \"kubernetes.io/projected/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed-kube-api-access-rvh74\") pod \"ingress-canary-pfqxj\" (UID: \"bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed\") " pod="openshift-ingress-canary/ingress-canary-pfqxj"
Oct 30 12:35:46.025456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.025327    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed-cert\") pod \"ingress-canary-pfqxj\" (UID: \"bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed\") " pod="openshift-ingress-canary/ingress-canary-pfqxj"
Oct 30 12:35:46.025456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.025366    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57060355-8a18-4a06-93bb-af1de546c2b7-tmp-dir\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:46.025456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.025393    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57060355-8a18-4a06-93bb-af1de546c2b7-config-volume\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:46.025456 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.025451    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655h6\" (UniqueName: \"kubernetes.io/projected/57060355-8a18-4a06-93bb-af1de546c2b7-kube-api-access-655h6\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:46.025647 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.025476    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:46.025647 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:46.025571    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:35:46.025647 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:46.025635    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls podName:57060355-8a18-4a06-93bb-af1de546c2b7 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:46.52561782 +0000 UTC m=+31.170605940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls") pod "dns-default-x5ck2" (UID: "57060355-8a18-4a06-93bb-af1de546c2b7") : secret "dns-default-metrics-tls" not found
Oct 30 12:35:46.025793 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.025701    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57060355-8a18-4a06-93bb-af1de546c2b7-tmp-dir\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:46.025985 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.025967    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57060355-8a18-4a06-93bb-af1de546c2b7-config-volume\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:46.029000 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.028981    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed-cert\") pod \"ingress-canary-pfqxj\" (UID: \"bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed\") " pod="openshift-ingress-canary/ingress-canary-pfqxj"
Oct 30 12:35:46.042268 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.042247    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvh74\" (UniqueName: \"kubernetes.io/projected/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed-kube-api-access-rvh74\") pod \"ingress-canary-pfqxj\" (UID: \"bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed\") " pod="openshift-ingress-canary/ingress-canary-pfqxj"
Oct 30 12:35:46.042410 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.042389    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-655h6\" (UniqueName: \"kubernetes.io/projected/57060355-8a18-4a06-93bb-af1de546c2b7-kube-api-access-655h6\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:46.146221 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.146196    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pfqxj"
Oct 30 12:35:46.501003 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.500931    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pfqxj"]
Oct 30 12:35:46.504540 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:46.504515    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd4c4a8_7ece_4ff4_b717_6556ec9c9aed.slice/crio-c7246937933fd61fc0af3b036e8d51a07f0f4411bc21b618b5a4e0e2935091aa WatchSource:0}: Error finding container c7246937933fd61fc0af3b036e8d51a07f0f4411bc21b618b5a4e0e2935091aa: Status 404 returned error can't find the container with id c7246937933fd61fc0af3b036e8d51a07f0f4411bc21b618b5a4e0e2935091aa
Oct 30 12:35:46.529438 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.529413    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:46.529531 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:46.529507    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:35:46.529574 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:46.529559    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls podName:57060355-8a18-4a06-93bb-af1de546c2b7 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:47.529545775 +0000 UTC m=+32.174533892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls") pod "dns-default-x5ck2" (UID: "57060355-8a18-4a06-93bb-af1de546c2b7") : secret "dns-default-metrics-tls" not found
Oct 30 12:35:46.846629 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.846601    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:46.846769 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.846673    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:46.846769 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.846763    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:46.859131 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.859096    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:35:46.859250 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.859159    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t6s7j\""
Oct 30 12:35:46.868667 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.868647    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:35:46.868667 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.868658    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:35:46.868840 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.868689    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bg2k7\""
Oct 30 12:35:46.868840 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.868716    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-54vc4\""
Oct 30 12:35:46.868918 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:46.868840    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:35:47.108078 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:47.108004    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pfqxj" event={"ID":"bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed","Type":"ContainerStarted","Data":"c7246937933fd61fc0af3b036e8d51a07f0f4411bc21b618b5a4e0e2935091aa"}
Oct 30 12:35:47.535712 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:47.535677    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:47.535874 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:47.535808    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:35:47.535874 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:47.535869    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls podName:57060355-8a18-4a06-93bb-af1de546c2b7 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:49.535850623 +0000 UTC m=+34.180838741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls") pod "dns-default-x5ck2" (UID: "57060355-8a18-4a06-93bb-af1de546c2b7") : secret "dns-default-metrics-tls" not found
Oct 30 12:35:48.905112 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:48.905077    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:48.982510 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:48.982479    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:49.550723 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:49.550689    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:49.550868 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:49.550805    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:35:49.550907 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:49.550900    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls podName:57060355-8a18-4a06-93bb-af1de546c2b7 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:53.550871838 +0000 UTC m=+38.195859977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls") pod "dns-default-x5ck2" (UID: "57060355-8a18-4a06-93bb-af1de546c2b7") : secret "dns-default-metrics-tls" not found
Oct 30 12:35:49.650997 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:49.650940    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:49.650997 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:49.650975    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:49.651129 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:49.651004    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m442\" (UniqueName: \"kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442\") pod \"network-check-target-rqz8x\" (UID: \"b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3\") " pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:49.651129 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:49.651082    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:35:49.651187 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:49.651140    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs podName:4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd nodeName:}" failed. No retries permitted until 2026-10-30 12:36:21.651125542 +0000 UTC m=+66.296113660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs") pod "network-metrics-daemon-crnn8" (UID: "4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd") : secret "metrics-daemon-secret" not found
Oct 30 12:35:49.653209 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:49.653185    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:49.653623 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:49.653604    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m442\" (UniqueName: \"kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442\") pod \"network-check-target-rqz8x\" (UID: \"b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3\") " pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:49.894366 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:49.894340    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:49.912510 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:49.912469    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:50.116285 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:50.116249    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pfqxj" event={"ID":"bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed","Type":"ContainerStarted","Data":"8f325ee6ca6a81661caca47372c76508a6ae8f87ab65389a18623c6b2942fc4c"}
Oct 30 12:35:50.148640 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:50.148598    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pfqxj" podStartSLOduration=2.6105773770000003 podStartE2EDuration="5.148567262s" totalImagesPullingTime="2.537989885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:50.147229666 +0000 UTC m=+34.792217807" watchObservedRunningTime="2026-10-30 12:35:50.148567262 +0000 UTC m=+34.793555400"
Oct 30 12:35:50.235695 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:50.235634    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rqz8x"]
Oct 30 12:35:50.238858 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:50.238822    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cc4ae3_3f1f_4bb5_9dfb_765f25583cb3.slice/crio-06c0bf196a568eee3f5ac271faeffeffe9f7855348c747a1d35224df2c0a6a77 WatchSource:0}: Error finding container 06c0bf196a568eee3f5ac271faeffeffe9f7855348c747a1d35224df2c0a6a77: Status 404 returned error can't find the container with id 06c0bf196a568eee3f5ac271faeffeffe9f7855348c747a1d35224df2c0a6a77
Oct 30 12:35:50.262091 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:50.262072    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9lfhr"]
Oct 30 12:35:50.264881 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:50.264851    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c94f5d7_3c75_41bd_963f_a6fffa86747a.slice/crio-695de77cd8b966d883b6db5327bed2f7020a5836fb167fdc051c1d022c91a65b WatchSource:0}: Error finding container 695de77cd8b966d883b6db5327bed2f7020a5836fb167fdc051c1d022c91a65b: Status 404 returned error can't find the container with id 695de77cd8b966d883b6db5327bed2f7020a5836fb167fdc051c1d022c91a65b
Oct 30 12:35:51.044775 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.044739    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p5vwm"]
Oct 30 12:35:51.062801 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.062775    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.064987 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.064956    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p5vwm"]
Oct 30 12:35:51.075319 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.075299    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t86w7\""
Oct 30 12:35:51.075429 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.075334    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 12:35:51.075429 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.075421    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 12:35:51.075538 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.075464    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 12:35:51.084116 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.084084    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 12:35:51.117708 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.117681    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-x8599-trltf"]
Oct 30 12:35:51.126690 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.126661    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:51.138671 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.138651    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-x8599-trltf"]
Oct 30 12:35:51.138792 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.138686    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9lfhr" event={"ID":"8c94f5d7-3c75-41bd-963f-a6fffa86747a","Type":"ContainerStarted","Data":"695de77cd8b966d883b6db5327bed2f7020a5836fb167fdc051c1d022c91a65b"}
Oct 30 12:35:51.138792 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.138711    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rqz8x" event={"ID":"b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3","Type":"ContainerStarted","Data":"06c0bf196a568eee3f5ac271faeffeffe9f7855348c747a1d35224df2c0a6a77"}
Oct 30 12:35:51.138792 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.138775    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.162026 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.162002    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2650cc31-aa41-466b-a61f-fd67f732ea03-data-volume\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.162132 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.162040    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.162132 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.162067    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.162215 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.162137    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6d6v\" (UniqueName: \"kubernetes.io/projected/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-api-access-g6d6v\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.162255 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.162212    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2650cc31-aa41-466b-a61f-fd67f732ea03-crio-socket\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.172674 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.172635    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 12:35:51.172839 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.172742    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9zjv6\""
Oct 30 12:35:51.192157 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.192134    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 12:35:51.234267 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.234234    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:51.263380 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.263337    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.263485 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.263394    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.263485 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.263420    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6d6v\" (UniqueName: \"kubernetes.io/projected/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-api-access-g6d6v\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.263485 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.263458    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-archives-path\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.263651 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.263513    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-trusted-ca-bundle\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.263651 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.263538    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2650cc31-aa41-466b-a61f-fd67f732ea03-crio-socket\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.263651 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.263555    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-service-ca-bundle\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.263651 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.263574    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pwxq\" (UniqueName: \"kubernetes.io/projected/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-kube-api-access-4pwxq\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.263651 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.263626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2650cc31-aa41-466b-a61f-fd67f732ea03-data-volume\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.263917 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.263899    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2650cc31-aa41-466b-a61f-fd67f732ea03-data-volume\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.264188 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:51.264171    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:51.264261 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:51.264219    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls podName:2650cc31-aa41-466b-a61f-fd67f732ea03 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:51.764205946 +0000 UTC m=+36.409194076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p5vwm" (UID: "2650cc31-aa41-466b-a61f-fd67f732ea03") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:51.264554 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.264535    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2650cc31-aa41-466b-a61f-fd67f732ea03-crio-socket\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.264629 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:51.264573    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 12:35:51.264682 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:51.264629    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm podName:2650cc31-aa41-466b-a61f-fd67f732ea03 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:51.764612023 +0000 UTC m=+36.409600157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p5vwm" (UID: "2650cc31-aa41-466b-a61f-fd67f732ea03") : configmap "kube-rbac-proxy" not found
Oct 30 12:35:51.283600 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.283570    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6d6v\" (UniqueName: \"kubernetes.io/projected/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-api-access-g6d6v\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.364252 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.364188    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-archives-path\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.364252 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.364243    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-trusted-ca-bundle\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.364437 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.364270    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-service-ca-bundle\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.364437 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.364294    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pwxq\" (UniqueName: \"kubernetes.io/projected/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-kube-api-access-4pwxq\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.364666 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.364610    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-archives-path\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.365201 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.365156    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-service-ca-bundle\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.365382 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.365342    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-trusted-ca-bundle\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.380782 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.380758    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pwxq\" (UniqueName: \"kubernetes.io/projected/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-kube-api-access-4pwxq\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.463893 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.463866    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:51.767620 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.767542    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.767787 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:51.767680    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 12:35:51.767787 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:51.767753    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm podName:2650cc31-aa41-466b-a61f-fd67f732ea03 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:52.767732656 +0000 UTC m=+37.412720787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p5vwm" (UID: "2650cc31-aa41-466b-a61f-fd67f732ea03") : configmap "kube-rbac-proxy" not found
Oct 30 12:35:51.767787 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:51.767771    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:51.767950 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:51.767811    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls podName:2650cc31-aa41-466b-a61f-fd67f732ea03 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:52.767800447 +0000 UTC m=+37.412788586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p5vwm" (UID: "2650cc31-aa41-466b-a61f-fd67f732ea03") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:51.767950 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.767680    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:51.786924 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.786893    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-x8599-trltf"]
Oct 30 12:35:51.791552 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:51.791524    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8749c6a2_8a38_412c_9ea0_ed31e4a1f072.slice/crio-0c47e7f7f5c18769d4e0b23be2c17147ba1aee8e6477f845a42903b4704b1b8e WatchSource:0}: Error finding container 0c47e7f7f5c18769d4e0b23be2c17147ba1aee8e6477f845a42903b4704b1b8e: Status 404 returned error can't find the container with id 0c47e7f7f5c18769d4e0b23be2c17147ba1aee8e6477f845a42903b4704b1b8e
Oct 30 12:35:51.844570 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.844503    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:51.944065 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:51.944010    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:52.094623 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.094589    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:52.126552 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.126512    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x8599-trltf" event={"ID":"8749c6a2-8a38-412c-9ea0-ed31e4a1f072","Type":"ContainerStarted","Data":"0c47e7f7f5c18769d4e0b23be2c17147ba1aee8e6477f845a42903b4704b1b8e"}
Oct 30 12:35:52.133458 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.133429    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf"]
Oct 30 12:35:52.171248 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.171215    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf"]
Oct 30 12:35:52.171399 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.171321    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf"
Oct 30 12:35:52.197554 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.197529    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:52.225853 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.225820    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xb6n4"]
Oct 30 12:35:52.244425 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.244405    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:52.248668 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.248642    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xb6n4"]
Oct 30 12:35:52.271150 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.271120    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8mjc\" (UniqueName: \"kubernetes.io/projected/7df7d7d2-f38d-4fef-8012-ef4a8c201277-kube-api-access-z8mjc\") pod \"migrator-66df5d496b-24kzf\" (UID: \"7df7d7d2-f38d-4fef-8012-ef4a8c201277\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf"
Oct 30 12:35:52.277198 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.277176    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2mbkm\""
Oct 30 12:35:52.277295 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.277283    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 12:35:52.277437 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.277412    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 12:35:52.277536 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.277497    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 12:35:52.277589 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.277575    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 12:35:52.277697 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.277683    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 12:35:52.277808 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.277793    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 12:35:52.277929 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.277915    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xkwtr\""
Oct 30 12:35:52.280667 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.280644    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:52.358860 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.358744    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:52.371846 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.371811    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/12bc2302-2423-4450-aeac-e30bd3c55ac9-signing-key\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:52.371963 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.371857    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/12bc2302-2423-4450-aeac-e30bd3c55ac9-signing-cabundle\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:52.371963 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.371877    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xdt2\" (UniqueName: \"kubernetes.io/projected/12bc2302-2423-4450-aeac-e30bd3c55ac9-kube-api-access-9xdt2\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:52.372059 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.371974    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8mjc\" (UniqueName: \"kubernetes.io/projected/7df7d7d2-f38d-4fef-8012-ef4a8c201277-kube-api-access-z8mjc\") pod \"migrator-66df5d496b-24kzf\" (UID: \"7df7d7d2-f38d-4fef-8012-ef4a8c201277\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf"
Oct 30 12:35:52.390585 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.390538    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8mjc\" (UniqueName: \"kubernetes.io/projected/7df7d7d2-f38d-4fef-8012-ef4a8c201277-kube-api-access-z8mjc\") pod \"migrator-66df5d496b-24kzf\" (UID: \"7df7d7d2-f38d-4fef-8012-ef4a8c201277\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf"
Oct 30 12:35:52.472848 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.472820    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/12bc2302-2423-4450-aeac-e30bd3c55ac9-signing-key\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:52.472972 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.472861    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/12bc2302-2423-4450-aeac-e30bd3c55ac9-signing-cabundle\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:52.472972 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.472883    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdt2\" (UniqueName: \"kubernetes.io/projected/12bc2302-2423-4450-aeac-e30bd3c55ac9-kube-api-access-9xdt2\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:52.474410 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.474389    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/12bc2302-2423-4450-aeac-e30bd3c55ac9-signing-cabundle\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:52.475974 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.475954    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/12bc2302-2423-4450-aeac-e30bd3c55ac9-signing-key\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:52.490387 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.490325    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdt2\" (UniqueName: \"kubernetes.io/projected/12bc2302-2423-4450-aeac-e30bd3c55ac9-kube-api-access-9xdt2\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:52.501171 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.501146    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf"
Oct 30 12:35:52.571380 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.571236    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:52.666102 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.666081    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:52.775006 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.774972    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:52.775160 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.775026    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:52.775160 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:52.775105    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 12:35:52.775160 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:52.775139    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:52.775302 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:52.775166    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm podName:2650cc31-aa41-466b-a61f-fd67f732ea03 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:54.775147861 +0000 UTC m=+39.420135981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p5vwm" (UID: "2650cc31-aa41-466b-a61f-fd67f732ea03") : configmap "kube-rbac-proxy" not found
Oct 30 12:35:52.775302 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:52.775204    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls podName:2650cc31-aa41-466b-a61f-fd67f732ea03 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:54.775185516 +0000 UTC m=+39.420173636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p5vwm" (UID: "2650cc31-aa41-466b-a61f-fd67f732ea03") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:52.780570 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.780544    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:52.823339 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.823315    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf"]
Oct 30 12:35:52.827123 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:52.827093    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df7d7d2_f38d_4fef_8012_ef4a8c201277.slice/crio-fc4460c82fc8d11ec6bca9a314edd9c4585dcce6958fbd9a3924946bdfe514cf WatchSource:0}: Error finding container fc4460c82fc8d11ec6bca9a314edd9c4585dcce6958fbd9a3924946bdfe514cf: Status 404 returned error can't find the container with id fc4460c82fc8d11ec6bca9a314edd9c4585dcce6958fbd9a3924946bdfe514cf
Oct 30 12:35:52.906392 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.906263    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xb6n4"]
Oct 30 12:35:52.910053 ip-10-0-129-133 kubenswrapper[2573]: W1030 12:35:52.910024    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bc2302_2423_4450_aeac_e30bd3c55ac9.slice/crio-e2e6a21dff10c43d7c3f272ac8438ff7b82907759453113e53fa2e0d8a5f9272 WatchSource:0}: Error finding container e2e6a21dff10c43d7c3f272ac8438ff7b82907759453113e53fa2e0d8a5f9272: Status 404 returned error can't find the container with id e2e6a21dff10c43d7c3f272ac8438ff7b82907759453113e53fa2e0d8a5f9272
Oct 30 12:35:52.955889 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:52.955814    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:53.130909 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:53.130875    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf" event={"ID":"7df7d7d2-f38d-4fef-8012-ef4a8c201277","Type":"ContainerStarted","Data":"fc4460c82fc8d11ec6bca9a314edd9c4585dcce6958fbd9a3924946bdfe514cf"}
Oct 30 12:35:53.132468 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:53.132413    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4" event={"ID":"12bc2302-2423-4450-aeac-e30bd3c55ac9","Type":"ContainerStarted","Data":"e2e6a21dff10c43d7c3f272ac8438ff7b82907759453113e53fa2e0d8a5f9272"}
Oct 30 12:35:53.581657 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:53.581619    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:53.581901 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:53.581763    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:35:53.581901 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:53.581823    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls podName:57060355-8a18-4a06-93bb-af1de546c2b7 nodeName:}" failed. No retries permitted until 2026-10-30 12:36:01.581805395 +0000 UTC m=+46.226793516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls") pod "dns-default-x5ck2" (UID: "57060355-8a18-4a06-93bb-af1de546c2b7") : secret "dns-default-metrics-tls" not found
Oct 30 12:35:54.790462 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:54.790424    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:54.790916 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:54.790477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:54.790916 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:54.790613    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:54.790916 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:54.790635    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 12:35:54.790916 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:54.790683    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls podName:2650cc31-aa41-466b-a61f-fd67f732ea03 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:58.790665117 +0000 UTC m=+43.435653250 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p5vwm" (UID: "2650cc31-aa41-466b-a61f-fd67f732ea03") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:54.790916 ip-10-0-129-133 kubenswrapper[2573]: E1030 12:35:54.790707    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm podName:2650cc31-aa41-466b-a61f-fd67f732ea03 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:58.790696399 +0000 UTC m=+43.435684519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p5vwm" (UID: "2650cc31-aa41-466b-a61f-fd67f732ea03") : configmap "kube-rbac-proxy" not found
Oct 30 12:35:55.141212 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:55.140339    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rqz8x" event={"ID":"b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3","Type":"ContainerStarted","Data":"6a8086f7495cedba35ce43cad88ed6bfca0b41045e09ad1ce54e03c171808175"}
Oct 30 12:35:55.141212 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:55.141109    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:55.143116 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:55.143091    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf" event={"ID":"7df7d7d2-f38d-4fef-8012-ef4a8c201277","Type":"ContainerStarted","Data":"a91cef28bd1ec9a439bbcfc405aedb16171fc87282f64fa869688e799f33b503"}
Oct 30 12:35:55.144738 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:55.144690    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x8599-trltf" event={"ID":"8749c6a2-8a38-412c-9ea0-ed31e4a1f072","Type":"ContainerStarted","Data":"81d7cacca8ee7bb99299d1f5af39274c44ecbd5cf6d9e27c80ea0112b8e300fd"}
Oct 30 12:35:55.173879 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:55.172862    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rqz8x" podStartSLOduration=35.112481438 podStartE2EDuration="39.172844349s" totalImagesPullingTime="4.060362911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:55.171834475 +0000 UTC m=+39.816822619" watchObservedRunningTime="2026-10-30 12:35:55.172844349 +0000 UTC m=+39.817832490"
Oct 30 12:35:55.204375 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:55.204313    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-x8599-trltf" podStartSLOduration=1.151321446 podStartE2EDuration="4.204295325s" totalImagesPullingTime="3.052973879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:55.203418807 +0000 UTC m=+39.848406946" watchObservedRunningTime="2026-10-30 12:35:55.204295325 +0000 UTC m=+39.849283464"
Oct 30 12:35:55.235509 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:55.235452    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf" podStartSLOduration=1.22696802 podStartE2EDuration="3.235440768s" totalImagesPullingTime="2.008472748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:55.234817753 +0000 UTC m=+39.879805892" watchObservedRunningTime="2026-10-30 12:35:55.235440768 +0000 UTC m=+39.880428908"
Oct 30 12:35:55.684582 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:55.684543    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:56.068386 ip-10-0-129-133 kubenswrapper[2573]: I1030 12:35:56.068338    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:56.129466 ip-10-0-129-133 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 12:35:56.145709 ip-10-0-129-133 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 12:35:56.145870 ip-10-0-129-133 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 12:35:56.145919 ip-10-0-129-133 systemd[1]: kubelet.service: Consumed 2.339s CPU time.
Oct 30 12:35:56.155808 ip-10-0-129-133 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:35:56.241330 ip-10-0-129-133 kubenswrapper[4522]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:35:56.241330 ip-10-0-129-133 kubenswrapper[4522]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:35:56.241330 ip-10-0-129-133 kubenswrapper[4522]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:35:56.241330 ip-10-0-129-133 kubenswrapper[4522]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:35:56.243332 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243263    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:56.243332 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243331    4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:56.243332 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243336    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243340    4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243344    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243347    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243366    4522 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243369    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243372    4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243374    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243377    4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243380    4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243382    4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243385    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243388    4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243390    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243393    4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243402    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243405    4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243407    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243410    4522 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243413    4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:56.243446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243415    4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243418    4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243421    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243425    4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243429    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243432    4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243435    4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243438    4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243440    4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243443    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243446    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243449    4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243452    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243455    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243457    4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243460    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243463    4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243466    4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243468    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243471    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:56.243926 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243474    4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243477    4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243480    4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243482    4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243484    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243488    4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243492    4522 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243495    4522 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243499    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243501    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243504    4522 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243506    4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243511    4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243514    4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243517    4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243519    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243522    4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243524    4522 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243527    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:56.244683 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243530    4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243532    4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243535    4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243538    4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243541    4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243543    4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243546    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243548    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243551    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243554    4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243556    4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243558    4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243561    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243564    4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243566    4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243569    4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243571    4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243574    4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243576    4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:56.245291 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243579    4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243582    4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243584    4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243587    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243590    4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243592    4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243596    4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243600    4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243602    4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243605    4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243607    4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243610    4522 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243613    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243616    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243618    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243621    4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243624    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243626    4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243629    4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243633    4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:56.245772 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243635    4522 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243638    4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243641    4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243643    4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243646    4522 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.243648    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245612    4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245628    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245636    4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245640    4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245643    4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245647    4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245650    4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245653    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245656    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245659    4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245661    4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245664    4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245667    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:56.246257 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245669    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245672    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245674    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245677    4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245680    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245682    4522 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245685    4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245688    4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245691    4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245694    4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245696    4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245699    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245702    4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245704    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245707    4522 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245710    4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245712    4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245715    4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245718    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:56.246736 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245720    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245725    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245728    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245731    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245733    4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245736    4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245739    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245743    4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245747    4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245749    4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245752    4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245754    4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245757    4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245760    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245762    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245765    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245767    4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245770    4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245773    4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:56.247227 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245776    4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245779    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245782    4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245785    4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245788    4522 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245790    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245793    4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245795    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245798    4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245800    4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245803    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245806    4522 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245808    4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245811    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245818    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245820    4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245823    4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245826    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245828    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245831    4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:56.247695 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245834    4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245836    4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245839    4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245842    4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245844    4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245847    4522 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245850    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245852    4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245855    4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245858    4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245860    4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245863    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245866    4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245869    4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245871    4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245874    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245877    4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245879    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245881    4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245884    4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:56.248173 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245886    4522 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245889    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245891    4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245894    4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245897    4522 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245899    4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245904    4522 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245907    4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245910    4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245912    4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245915    4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245918    4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245920    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245924    4522 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.245927    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246424    4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246431    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246435    4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246438    4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246441    4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:56.248675 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246444    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246447    4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246449    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246452    4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246455    4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246458    4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246461    4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246463    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246466    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246469    4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246471    4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246474    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246476    4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246479    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246481    4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246484    4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246487    4522 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246489    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246495    4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:56.249162 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246497    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246500    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246503    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246505    4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246508    4522 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246511    4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246513    4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246516    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246518    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246521    4522 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246524    4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246526    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246529    4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246532    4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246534    4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246537    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246539    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246542    4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246545    4522 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246547    4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:56.249636 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246550    4522 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246553    4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246555    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246558    4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246560    4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246563    4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246566    4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246569    4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246573    4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246576    4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246579    4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246585    4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246588    4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246590    4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246593    4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246595    4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246598    4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246601    4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246604    4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:56.250287 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246606    4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246609    4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246611    4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246614    4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246617    4522 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246619    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246622    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246624    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246627    4522 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246629    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246632    4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246634    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246637    4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246640    4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246642    4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246645    4522 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246648    4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246650    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246652    4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246655    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:56.251131 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246658    4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246660    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246663    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246665    4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246670    4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246673    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246675    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246678    4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246680    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246683    4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246686    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246688    4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246692    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246694    4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246697    4522 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246699    4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246702    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246705    4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246707    4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:56.252021 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246710    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246712    4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246715    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.246717    4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246807    4522 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246817    4522 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246827    4522 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246834    4522 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246852    4522 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246858    4522 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246862    4522 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246866    4522 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246870    4522 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246873    4522 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246876    4522 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246880    4522 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246883    4522 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246887    4522 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246890    4522 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246893    4522 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246896    4522 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246899    4522 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246903    4522 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246906    4522 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:35:56.252987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246909    4522 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246912    4522 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246915    4522 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246919    4522 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246922    4522 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246925    4522 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246928    4522 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246931    4522 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246934    4522 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246937    4522 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246940    4522 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246943    4522 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246948    4522 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246951    4522 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246954    4522 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246957    4522 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246960    4522 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246963    4522 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246968    4522 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246971    4522 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246974    4522 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246977    4522 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246980    4522 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246984    4522 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:35:56.254035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246987    4522 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246991    4522 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246994    4522 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.246997    4522 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247001    4522 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247004    4522 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247007    4522 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247010    4522 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247013    4522 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247016    4522 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247020    4522 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247023    4522 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247026    4522 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247030    4522 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247033    4522 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247036    4522 flags.go:64] FLAG: --help="false"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247039    4522 flags.go:64] FLAG: --hostname-override="ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247043    4522 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247046    4522 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247049    4522 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247052    4522 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247056    4522 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247059    4522 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:35:56.255124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247062    4522 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247066    4522 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247069    4522 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247072    4522 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247075    4522 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247078    4522 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247081    4522 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247084    4522 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247087    4522 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247090    4522 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247093    4522 flags.go:64] FLAG: --lock-file=""
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247096    4522 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247099    4522 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247102    4522 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247108    4522 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247110    4522 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247113    4522 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247116    4522 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247119    4522 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247123    4522 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247126    4522 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247129    4522 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247133    4522 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247136    4522 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247140    4522 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:35:56.256145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247143    4522 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247146    4522 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247149    4522 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247152    4522 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247155    4522 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247158    4522 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247161    4522 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247168    4522 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247171    4522 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247175    4522 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247178    4522 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247181    4522 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247184    4522 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247187    4522 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247190    4522 flags.go:64] FLAG: --port="10250"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247193    4522 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247196    4522 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-034db7daabe8ccabb"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247199    4522 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247202    4522 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247205    4522 flags.go:64] FLAG: --register-node="true"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247208    4522 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247211    4522 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247214    4522 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247217    4522 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247220    4522 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247224    4522 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:35:56.257222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247227    4522 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247230    4522 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247233    4522 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247236    4522 flags.go:64] FLAG: --runonce="false"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247239    4522 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247244    4522 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247248    4522 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247250    4522 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247254    4522 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247256    4522 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247260    4522 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247263    4522 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247266    4522 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247269    4522 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247271    4522 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247274    4522 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247278    4522 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247281    4522 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247284    4522 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247290    4522 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247294    4522 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247297    4522 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247301    4522 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247304    4522 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247307    4522 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:35:56.258394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247310    4522 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247313    4522 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247316    4522 flags.go:64] FLAG: --v="2"
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247320    4522 flags.go:64] FLAG: --version="false"
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247324    4522 flags.go:64] FLAG: --vmodule=""
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247328    4522 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247331    4522 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247475    4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247479    4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247482    4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247484    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247487    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247491    4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247494    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247497    4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247500    4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247503    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247506    4522 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247508    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247511    4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247513    4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:56.259277 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247517    4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247519    4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247522    4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247525    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247528    4522 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247534    4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247538    4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247541    4522 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247544    4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247546    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247549    4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247552    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247555    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247558    4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247560    4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247563    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247566    4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247568    4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247571    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:56.260018 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247574    4522 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247576    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247579    4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247582    4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247586    4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247589    4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247591    4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247594    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247597    4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247600    4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247602    4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247605    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247607    4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247610    4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247613    4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247615    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247618    4522 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247621    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247624    4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:56.260598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247627    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247630    4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247632    4522 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247635    4522 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247638    4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247640    4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247643    4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247646    4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247648    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247651    4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247653    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247656    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247659    4522 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247661    4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247664    4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247666    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247669    4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247673    4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247675    4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247678    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:56.261093 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247680    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247683    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247685    4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247688    4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247691    4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247694    4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247696    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247699    4522 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247702    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247704    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247707    4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247712    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247714    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247717    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247719    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247722    4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247725    4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247727    4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247730    4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247732    4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:56.261608 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247735    4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247737    4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247742    4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247746    4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247749    4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247753    4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247756    4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247758    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247761    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247765    4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247768    4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247770    4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247773    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247776    4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.247802    4522 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:35:56.262153 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247836    4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247840    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247843    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247846    4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247849    4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247852    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247855    4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247857    4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247860    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247864    4522 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247867    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247869    4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247872    4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247874    4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247877    4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247880    4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247883    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247885    4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247888    4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247891    4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:56.262626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247893    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247896    4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247898    4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247901    4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247903    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247906    4522 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247910    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247913    4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247916    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247918    4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247921    4522 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247923    4522 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247926    4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247928    4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247931    4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247934    4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247936    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247939    4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247942    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247944    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:56.263184 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247947    4522 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247951    4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247954    4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247957    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247960    4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247962    4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247965    4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247967    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247970    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247972    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247975    4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247978    4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247980    4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247983    4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247986    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247988    4522 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247992    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247994    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.247999    4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:56.263698 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248001    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248004    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248007    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248009    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248011    4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248014    4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248017    4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248020    4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248022    4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248025    4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248028    4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248032    4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248036    4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248039    4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248044    4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248046    4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248049    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248052    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248055    4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:56.264323 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248058    4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248061    4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248063    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248066    4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248068    4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248071    4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248073    4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248076    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248079    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248081    4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248084    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248087    4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248091    4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248094    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248096    4522 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248099    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248101    4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248104    4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248107    4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248109    4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:56.265056 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248112    4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248114    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248117    4522 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248121    4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248124    4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248127    4522 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248130    4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.248134    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.252516    4522 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.252538    4522 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.252561    4522 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.252569    4522 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252648    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252656    4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252661    4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252666    4522 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252671    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252675    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252679    4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:56.265571 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252684    4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252688    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252693    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252697    4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252702    4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252706    4522 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252710    4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252714    4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252719    4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252723    4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252728    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252732    4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252737    4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252741    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252746    4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252751    4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252756    4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252760    4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252765    4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252770    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:56.266028 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252774    4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252780    4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252788    4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252794    4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252799    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252803    4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252808    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252812    4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252816    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252820    4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252825    4522 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252829    4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252834    4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252838    4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252842    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252847    4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252852    4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252857    4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252861    4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:56.266702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252867    4522 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252871    4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252875    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252879    4522 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252884    4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252888    4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252892    4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252896    4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252901    4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252905    4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252909    4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252914    4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252918    4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252922    4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252926    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252931    4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252936    4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252940    4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252945    4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:56.267196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252950    4522 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252954    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252959    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252963    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252967    4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252971    4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252976    4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252980    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252984    4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252988    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252993    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.252997    4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253001    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253008    4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253014    4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253020    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253025    4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253029    4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253034    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:56.267724 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253038    4522 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253042    4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253047    4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253051    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253055    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253059    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253063    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253068    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253073    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253077    4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253081    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253092    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253097    4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253101    4522 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253105    4522 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253110    4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253114    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253118    4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253123    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253127    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:56.268196 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253131    4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253135    4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.253177    4522 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253470    4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253480    4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253485    4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253489    4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253494    4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253498    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253503    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253507    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253511    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253515    4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253520    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253525    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253529    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:56.268706 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253533    4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253537    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253542    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253546    4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253550    4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253554    4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253558    4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253566    4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253573    4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253579    4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253583    4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253588    4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253592    4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253596    4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253601    4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253605    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253609    4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253613    4522 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253617    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:56.269105 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253622    4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253626    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253630    4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253634    4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253639    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253644    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253648    4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253652    4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253656    4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253661    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253665    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253669    4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253674    4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253678    4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253682    4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253687    4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253691    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253695    4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253699    4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253703    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:56.269577 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253708    4522 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253712    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253717    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253721    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253726    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253731    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253735    4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253739    4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253743    4522 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253748    4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253774    4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253780    4522 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253786    4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253790    4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253795    4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253799    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253807    4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253812    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253816    4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253820    4522 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:56.270051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253825    4522 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253829    4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253834    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253838    4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253842    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253847    4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253851    4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253855    4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253860    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253864    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253868    4522 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253872    4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253877    4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253881    4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253885    4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253890    4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253895    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253899    4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253903    4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253908    4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:56.270619 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253912    4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253916    4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253921    4522 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253925    4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253932    4522 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253936    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253941    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253945    4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253951    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253956    4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253960    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253965    4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253969    4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.253976    4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.254015    4522 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254066    4522 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:35:56.271119 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254074    4522 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254079    4522 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254084    4522 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254089    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254095    4522 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254099    4522 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254104    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254108    4522 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254113    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254118    4522 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254122    4522 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254127    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254132    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254137    4522 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254141    4522 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254145    4522 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254150    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254154    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254158    4522 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:35:56.271541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254163    4522 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254169    4522 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254174    4522 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254178    4522 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254183    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254187    4522 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254192    4522 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254197    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254202    4522 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254206    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254211    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254215    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254219    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254224    4522 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254229    4522 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254233    4522 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254237    4522 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254241    4522 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254245    4522 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254250    4522 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:35:56.272000 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254254    4522 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254259    4522 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254263    4522 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254267    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254271    4522 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254276    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254281    4522 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254285    4522 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254290    4522 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254294    4522 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254298    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254302    4522 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254307    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254317    4522 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254321    4522 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254325    4522 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254330    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254334    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254340    4522 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254345    4522 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:35:56.272559 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254367    4522 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254372    4522 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254376    4522 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254380    4522 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254384    4522 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254389    4522 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254393    4522 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254398    4522 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254402    4522 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254406    4522 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254410    4522 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254414    4522 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254419    4522 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254424    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254428    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254433    4522 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254437    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254441    4522 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254445    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254451    4522 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:35:56.273086 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254455    4522 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254461    4522 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254466    4522 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254471    4522 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254475    4522 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254482    4522 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254486    4522 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254491    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254495    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254499    4522 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254506    4522 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254511    4522 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254515    4522 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254520    4522 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254524    4522 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254528    4522 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254533    4522 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254537    4522 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254541    4522 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:35:56.273598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254545    4522 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254549    4522 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254554    4522 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254562    4522 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254568    4522 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254573    4522 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:35:56.254578    4522 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.254658    4522 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.256119    4522 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.256187    4522 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.256460    4522 server.go:1014] "Starting client certificate rotation"
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.256586    4522 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.256636    4522 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:30:16 +0000 UTC" deadline="2028-04-25 03:10:51.422276827 +0000 UTC"
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.256663    4522 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13022h34m55.165615638s"
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.257141    4522 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:35:56.274051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.258729    4522 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:35:56.274557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.260129    4522 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:35:56.274557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.261696    4522 log.go:25] "Validated CRI v1 image API"
Oct 30 12:35:56.274557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.262385    4522 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:35:56.274557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.263815    4522 fs.go:135] Filesystem UUIDs: map[073ad6aa-9489-412b-be5d-817070a18bf6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 acda0c59-d135-4c6e-ac7f-e77170b4b540:/dev/nvme0n1p3]
Oct 30 12:35:56.274673 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.263836    4522 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06c0bf196a568eee3f5ac271faeffeffe9f7855348c747a1d35224df2c0a6a77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06c0bf196a568eee3f5ac271faeffeffe9f7855348c747a1d35224df2c0a6a77/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c47e7f7f5c18769d4e0b23be2c17147ba1aee8e6477f845a42903b4704b1b8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c47e7f7f5c18769d4e0b23be2c17147ba1aee8e6477f845a42903b4704b1b8e/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19a7334ccbf0db5f2d47fe3f70e73ca8afa7540e29a1e9577d1f412e127c7055/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19a7334ccbf0db5f2d47fe3f70e73ca8afa7540e29a1e9577d1f412e127c7055/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37962d5f41d40956e0372be06ffe4fa326336ed3132a67a554e4855e51592964/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37962d5f41d40956e0372be06ffe4fa326336ed3132a67a554e4855e51592964/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66cf76fc27b029e8af3e440792091cf93c3a7ad4ca65b17c66d224468cc05d7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66cf76fc27b029e8af3e440792091cf93c3a7ad4ca65b17c66d224468cc05d7c/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/695de77cd8b966d883b6db5327bed2f7020a5836fb167fdc051c1d022c91a65b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/695de77cd8b966d883b6db5327bed2f7020a5836fb167fdc051c1d022c91a65b/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c29b7a0a904b247bd224e23897730fffa4f44bd1c25e2f70fb45888146c0773/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c29b7a0a904b247bd224e23897730fffa4f44bd1c25e2f70fb45888146c0773/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4da9269af03503d70d162191d4d482a146238bd08e3b19d753f366476d3c5cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4da9269af03503d70d162191d4d482a146238bd08e3b19d753f366476d3c5cf/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b36b5a2eed3f861a2d192395222ac197c11f9f212ec2b657ad28fcbc04709aa4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b36b5a2eed3f861a2d192395222ac197c11f9f212ec2b657ad28fcbc04709aa4/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc4ec294735d21991682e0d19ea309829f1b4ad0b3248110acc14ccfe80a101b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc4ec294735d21991682e0d19ea309829f1b4ad0b3248110acc14ccfe80a101b/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c504131be22c2c7ca2e34ce61ef4fb50a57b9edc5cb6e0064e554ba49424a67e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c504131be22c2c7ca2e34ce61ef4fb50a57b9edc5cb6e0064e554ba49424a67e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7246937933fd61fc0af3b036e8d51a07f0f4411bc21b618b5a4e0e2935091aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7246937933fd61fc0af3b036e8d51a07f0f4411bc21b618b5a4e0e2935091aa/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d289cf88870519d59981a96755cec88250957564a6861f4391dedf8d3d97f2ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d289cf88870519d59981a96755cec88250957564a6861f4391dedf8d3d97f2ab/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2e6a21dff10c43d7c3f272ac8438ff7b82907759453113e53fa2e0d8a5f9272/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2e6a21dff10c43d7c3f272ac8438ff7b82907759453113e53fa2e0d8a5f9272/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3245af77502ccbbbd098ab88a812a8132ac0eded9bd73dc334f4c1ae128688b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3245af77502ccbbbd098ab88a812a8132ac0eded9bd73dc334f4c1ae128688b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc4460c82fc8d11ec6bca9a314edd9c4585dcce6958fbd9a3924946bdfe514cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc4460c82fc8d11ec6bca9a314edd9c4585dcce6958fbd9a3924946bdfe514cf/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02a59e47-332d-40bb-af4a-2a0b59a816c8/volumes/kubernetes.io~projected/kube-api-access-h8s2g:{mountpoint:/var/lib/kubelet/pods/02a59e47-332d-40bb-af4a-2a0b59a816c8/volumes/kubernetes.io~projected/kube-api-access-h8s2g major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12bc2302-2423-4450-aeac-e30bd3c55ac9/volumes/kubernetes.io~projected/kube-api-access-9xdt2:{mountpoint:/var/lib/kubelet/pods/12bc2302-2423-4450-aeac-e30bd3c55ac9/volumes/kubernetes.io~projected/kube-api-access-9xdt2 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12bc2302-2423-4450-aeac-e30bd3c55ac9/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/12bc2302-2423-4450-aeac-e30bd3c55ac9/volumes/kubernetes.io~secret/signing-key major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2650cc31-aa41-466b-a61f-fd67f732ea03/volumes/kubernetes.io~projected/kube-api-access-g6d6v:{mountpoint:/var/lib/kubelet/pods/2650cc31-aa41-466b-a61f-fd67f732ea03/volumes/kubernetes.io~projected/kube-api-access-g6d6v major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d/volumes/kubernetes.io~projected/kube-api-access-cpvmw:{mountpoint:/var/lib/kubelet/pods/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d/volumes/kubernetes.io~projected/kube-api-access-cpvmw major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd/volumes/kubernetes.io~projected/kube-api-access-bnz5h:{mountpoint:/var/lib/kubelet/pods/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd/volumes/kubernetes.io~projected/kube-api-access-bnz5h major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b382eac-dd89-486d-b80b-1f5a402f23d4/volumes/kubernetes.io~projected/kube-api-access-dd6r5:{mountpoint:/var/lib/kubelet/pods/4b382eac-dd89-486d-b80b-1f5a402f23d4/volumes/kubernetes.io~projected/kube-api-access-dd6r5 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57060355-8a18-4a06-93bb-af1de546c2b7/volumes/kubernetes.io~projected/kube-api-access-655h6:{mountpoint:/var/lib/kubelet/pods/57060355-8a18-4a06-93bb-af1de546c2b7/volumes/kubernetes.io~projected/kube-api-access-655h6 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7991e362-d106-4607-b8ee-ad70dfff9e7d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7991e362-d106-4607-b8ee-ad70dfff9e7d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7991e362-d106-4607-b8ee-ad70dfff9e7d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7991e362-d106-4607-b8ee-ad70dfff9e7d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7991e362-d106-4607-b8ee-ad70dfff9e7d/volumes/kubernetes.io~projected/kube-api-access-jsbfd:{mountpoint:/var/lib/kubelet/pods/7991e362-d106-4607-b8ee-ad70dfff9e7d/volumes/kubernetes.io~projected/kube-api-access-jsbfd major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7df7d7d2-f38d-4fef-8012-ef4a8c201277/volumes/kubernetes.io~projected/kube-api-access-z8mjc:{mountpoint:/var/lib/kubelet/pods/7df7d7d2-f38d-4fef-8012-ef4a8c201277/volumes/kubernetes.io~projected/kube-api-access-z8mjc major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8749c6a2-8a38-412c-9ea0-ed31e4a1f072/volumes/kubernetes.io~projected/kube-api-access-4pwxq:{mountpoint:/var/lib/kubelet/pods/8749c6a2-8a38-412c-9ea0-ed31e4a1f072/volumes/kubernetes.io~projected/kube-api-access-4pwxq major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c94f5d7-3c75-41bd-963f-a6fffa86747a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8c94f5d7-3c75-41bd-963f-a6fffa86747a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e69cda5-58f4-409b-8575-d0d8f3c09ba6/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8e69cda5-58f4-409b-8575-d0d8f3c09ba6/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a695456e-fedc-42fb-955b-5c55f356b0c3/volumes/kubernetes.io~projected/kube-api-access-vnhcl:{mountpoint:/var/lib/kubelet/pods/a695456e-fedc-42fb-955b-5c55f356b0c3/volumes/kubernetes.io~projected/kube-api-access-vnhcl major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c/volumes/kubernetes.io~projected/kube-api-access-d5vcf:{mountpoint:/var/lib/kubelet/pods/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c/volumes/kubernetes.io~projected/kube-api-access-d5vcf major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3/volumes/kubernetes.io~projected/kube-api-access-2m442:{mountpoint:/var/lib/kubelet/pods/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3/volumes/kubernetes.io~projected/kube-api-access-2m442 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed/volumes/kubernetes.io~projected/kube-api-access-rvh74:{mountpoint:/var/lib/kubelet/pods/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed/volumes/kubernetes.io~projected/kube-api-access-rvh74 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d22f52c3-6029-4b19-aa6a-9843e569dbc4/volumes/kubernetes.io~projected/kube-api-access-prz95:{mountpoint:/var/lib/kubelet/pods/d22f52c3-6029-4b19-aa6a-9843e569dbc4/volumes/kubernetes.io~projected/kube-api-access-prz95 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e14fcd05-8067-4312-b9bb-90c4c9a15b23/volumes/kubernetes.io~projected/kube-api-access-kg67k:{mountpoint:/var/lib/kubelet/pods/e14fcd05-8067-4312-b9bb-90c4c9a15b23/volumes/kubernetes.io~projected/kube-api-access-kg67k major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0c44f33934e1d150c014b707e83daf7468856b850c7a065beb6d63d18f85a75e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6551d60551cfdc5543738fc15584fcc024af9ce030ccfd9bf107771367b0ecfb/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d74b41b466c81e62a738c713df812b631cf0525a33c8931f30999f598e219eb6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7cf60c9f0f67a6ef98601f80a9141ff193ddf5014ccfc0ca62119aed50536113/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9526f9d0e73a7780d8181b3db7abcfd928b04449465b5be5732dacd9eabc9aec/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/37eb0cdef836d4b8a7c424cae6c19985dbe1549e2ea5725dcfa3cee7af1d185a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d8a968ab44476a094a40406afb9c0769fab5458e04fff38c5634eba6093a78a7/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e56debc1c99f252be1c2b25eae4cc95ad482de4369fcea3bca182bf7110fee15/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/36dba525f0ab76f1f162e2d8baf24108b2ba39d605c968e5500f17b0013173ca/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/2b92e3462f6244060f28e60064a41235ccfa0a5a32639207c73034052118818d/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/bbdac3309df43516a092de17c158e54aa4dd6f2afbcaebb3490d940beab50e1c/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/d0b5ec9af7c9cac045f3360eedab9e9b2d130341f63287b6a5d15fb85499b9d4/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/eea8bd17412495735cf6a3c660266a7c5620fb8754b33e1153edaecf28c8b729/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7a079b4c3f4ab590f2a85cbdfca77f2fc0a506ddce41f39dc91b48e9a430e580/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/17c9d475bc58492e85520cb794e716e5e50375ceccd9418cbe8b77616387a95a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a3f3d2b063942c328da1a3863ff5486bd5d630ea97be0ba2f32bf748b2197e38/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c7fd0698d37d8dfa060c61b5fcd1e60de0e2a38a2e50242f9482ff93028873a4/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/317735b8937999a9a19596de310d0d5e8ef270a7cb74a0900d39d30e27a3baff/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/35ebd0843e38944eb95c349795deda34e7104d5fc177e8815b748b0861d1e5c2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/924de68e47c917ed9b72f0f97963a3856d2fc23be74ab4294d955b5764e2dfe9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/205dc2e86fd7a13b8b06b6a2c87bb04bc879a5ec27d3458c8b95e67e6ffd0034/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/c51ca624dbac310cc04b1bba463202f1a094eced9e06f940b75b25ea5ae1f103/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/63a59dd74b848cc45c4b171954a35a3658ab11a95cac1989eaaa278131c56a66/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/e21e3d6aa84fe24d15c9be4453cacadd536aa2267aa24ac3ff6a1d612e854091/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/318fd9dd9dcd4de0c24d69eb144baf659d489df6ac2333e6da9eba19497b2480/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/be287d973b1e84baaaf869ac680f671290a72b03eb58e43e29dc5f870f1eeacd/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/fcb0c78ca3ddde2c5f0549d706be5b34092bdf21c5da657ccfb87a4ab082528b/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/4f5381df1b47232af1e6b8cdfa34efd8757af8243408ecee34f6fcf4654c27ab/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/d431c98a139a9f6da2b2c4513876160df7bc9a2432e9be152a76cfa5b275a238/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/f116e5f342e4997377d57e49196cfa9b8d93821b03d0b441a0e338923b372dcd/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/01362948ca538ab110fb09cbd74084cf39e37091c836197bdac8b71f816b23a1/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/229b15efa9dfca2cdf0f39c3559cbd0b5ef6fc4ee50070f96d180e0972be4af0/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/110bdc3d657b05714ecd4252b1f1e8959a60af695654f69a13f7fb6f05c019e3/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/50bb60680126b75e24f185e19c90fb5bc93936ac0a53cd7dd7152848210c3edf/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/583c5c94dba9cf83d9f3e4cad47f7158d503b7cffa8ec3533520d68abb3b3f95/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/cc1dd8f9b848fb1fc09184203b0d53dd23e3e172ef63a87db9e20824630f9f47/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/68f3227bd7db7ea9af02784425913b0fc9ee10d0163426efbfcfabb823ece143/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ac6290dddc8fae0d78033c014eb6ae8d48869a64c89e135aa057ae2aaae063be/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/186ba414e01fa5f94f9a9535c00b27eea76618a3f36342cd19220b96ceeaf77f/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/ade871dfb854b3f8cb5673895db8639e8e44f1e9f41729a87ca02615cb90ff8b/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/2f6901199bdbc18b79c45c79360feaabe482242e9aeea88b27077ea9a92ae4b8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ce953f006a814288fe5ef6b014827ee4afd284c9fa174a9eac287f02b99fc34b/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.271039    4522 manager.go:217] Machine: {Timestamp:2026-10-30 12:35:56.270585185 +0000 UTC m=+0.085911665 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099121 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a5095fa5d468d0d88a55e83f5fc7c SystemUUID:ec2a5095-fa5d-468d-0d88-a55e83f5fc7c BootID:e170ef7b-bfbb-4c4f-9ddc-828084546ec3 Filesystems:[{Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3/volumes/kubernetes.io~projected/kube-api-access-2m442 DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2650cc31-aa41-466b-a61f-fd67f732ea03/volumes/kubernetes.io~projected/kube-api-access-g6d6v DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7991e362-d106-4607-b8ee-ad70dfff9e7d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd/volumes/kubernetes.io~projected/kube-api-access-bnz5h DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b36b5a2eed3f861a2d192395222ac197c11f9f212ec2b657ad28fcbc04709aa4/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/695de77cd8b966d883b6db5327bed2f7020a5836fb167fdc051c1d022c91a65b/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/02a59e47-332d-40bb-af4a-2a0b59a816c8/volumes/kubernetes.io~projected/kube-api-access-h8s2g DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c504131be22c2c7ca2e34ce61ef4fb50a57b9edc5cb6e0064e554ba49424a67e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c94f5d7-3c75-41bd-963f-a6fffa86747a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06c0bf196a568eee3f5ac271faeffeffe9f7855348c747a1d35224df2c0a6a77/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c47e7f7f5c18769d4e0b23be2c17147ba1aee8e6477f845a42903b4704b1b8e/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7df7d7d2-f38d-4fef-8012-ef4a8c201277/volumes/kubernetes.io~projected/kube-api-access-z8mjc DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57060355-8a18-4a06-93bb-af1de546c2b7/volumes/kubernetes.io~projected/kube-api-access-655h6 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12bc2302-2423-4450-aeac-e30bd3c55ac9/volumes/kubernetes.io~projected/kube-api-access-9xdt2 DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66cf76fc27b029e8af3e440792091cf93c3a7ad4ca65b17c66d224468cc05d7c/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c/volumes/kubernetes.io~projected/kube-api-access-d5vcf DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d/volumes/kubernetes.io~projected/kube-api-access-cpvmw DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12bc2302-2423-4450-aeac-e30bd3c55ac9/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4da9269af03503d70d162191d4d482a146238bd08e3b19d753f366476d3c5cf/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d22f52c3-6029-4b19-aa6a-9843e569dbc4/volumes/kubernetes.io~projected/kube-api-access-prz95 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b382eac-dd89-486d-b80b-1f5a402f23d4/volumes/kubernetes.io~projected/kube-api-access-dd6r5 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19a7334ccbf0db5f2d47fe3f70e73ca8afa7540e29a1e9577d1f412e127c7055/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc4460c82fc8d11ec6bca9a314edd9c4585dcce6958fbd9a3924946bdfe514cf/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c29b7a0a904b247bd224e23897730fffa4f44bd1c25e2f70fb45888146c0773/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7991e362-d106-4607-b8ee-ad70dfff9e7d/volumes/kubernetes.io~projected/kube-api-access-jsbfd DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3245af77502ccbbbd098ab88a812a8132ac0eded9bd73dc334f4c1ae128688b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d289cf88870519d59981a96755cec88250957564a6861f4391dedf8d3d97f2ab/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc4ec294735d21991682e0d19ea309829f1b4ad0b3248110acc14ccfe80a101b/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8749c6a2-8a38-412c-9ea0-ed31e4a1f072/volumes/kubernetes.io~projected/kube-api-access-4pwxq DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e69cda5-58f4-409b-8575-d0d8f3c09ba6/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7246937933fd61fc0af3b036e8d51a07f0f4411bc21b618b5a4e0e2935091aa/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37962d5f41d40956e0372be06ffe4fa326336ed3132a67a554e4855e51592964/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e14fcd05-8067-4312-b9bb-90c4c9a15b23/volumes/kubernetes.io~projected/kube-api-access-kg67k DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed/volumes/kubernetes.io~projected/kube-api-access-rvh74 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2e6a21dff10c43d7c3f272ac8438ff7b82907759453113e53fa2e0d8a5f9272/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7991e362-d106-4607-b8ee-ad70dfff9e7d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a695456e-fedc-42fb-955b-5c55f356b0c3/volumes/kubernetes.io~projected/kube-api-access-vnhcl DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:06c0bf196a568ee MacAddress:9a:be:8e:1f:af:7b Speed:10000 Mtu:8901} {Name:0c47e7f7f5c1876 MacAddress:6a:dc:48:2b:0f:56 Speed:10000 Mtu:8901} {Name:695de77cd8b966d MacAddress:02:1d:90:e9:66:f4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c7:80:30:e3 Speed:0 Mtu:9001} {Name:br-int MacAddress:3a:72:6f:af:0d:27 Speed:0 Mtu:8901} {Name:c7246937933fd61 MacAddress:fe:9c:7b:ca:93:f9 Speed:10000 Mtu:8901} {Name:e2e6a21dff10c43 MacAddress:06:ac:9c:87:d6:6a Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c7:80:30:e3 Speed:0 Mtu:9001} {Name:fc4460c82fc8d11 MacAddress:02:63:a2:ca:93:12 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:4a:dc:ab:f6:a9:3e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4a:b3:1b:83:f5:35 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.271284    4522 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.271321    4522 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.271591    4522 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.271615    4522 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-133.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.271729    4522 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.271737    4522 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.271747    4522 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.271765    4522 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.271843    4522 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.271990    4522 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.272092    4522 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.272125    4522 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.272142    4522 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.272152    4522 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.272160    4522 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.272168    4522 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.272900    4522 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.272914    4522 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:35:56.274825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.273429    4522 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:35:56.284929 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.284912    4522 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:35:56.284929 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.284925    4522 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:35:56.285060 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285049    4522 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 12:35:56.285243 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285232    4522 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:35:56.285272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285262    4522 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:35:56.285427 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285413    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:35:56.285459 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285441    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:35:56.285459 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285456    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:35:56.285509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285464    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:35:56.285509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285471    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:35:56.285509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285477    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:35:56.285509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285482    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:35:56.285509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285488    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:35:56.285509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285494    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:35:56.285509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285508    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:35:56.285732 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285521    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:35:56.285732 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285530    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:35:56.285732 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285559    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:35:56.285732 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.285567    4522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:35:56.289091 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.289073    4522 server.go:1263] "Started kubelet"
Oct 30 12:35:56.289186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.289136    4522 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:35:56.289650 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.289435    4522 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:35:56.289701 ip-10-0-129-133 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:35:56.289873 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.289818    4522 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:35:56.290262 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.290225    4522 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:35:56.290950 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.290937    4522 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:35:56.294581 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.294564    4522 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:35:56.295975 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.294741    4522 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:35:56.295975 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.294789    4522 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:30:18 +0000 UTC" deadline="2028-07-13 12:40:58.107088608 +0000 UTC"
Oct 30 12:35:56.295975 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.294820    4522 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:35:56.295975 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.294824    4522 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14928h5m1.812268547s"
Oct 30 12:35:56.295975 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.294832    4522 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:35:56.295975 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:56.294805    4522 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-133.ec2.internal\" not found"
Oct 30 12:35:56.295975 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.294870    4522 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:35:56.296536 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296487    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02a59e47-332d-40bb-af4a-2a0b59a816c8" volumeName="kubernetes.io/projected/02a59e47-332d-40bb-af4a-2a0b59a816c8-kube-api-access-h8s2g" seLinuxMountContext=""
Oct 30 12:35:56.296536 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296534    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57060355-8a18-4a06-93bb-af1de546c2b7" volumeName="kubernetes.io/empty-dir/57060355-8a18-4a06-93bb-af1de546c2b7-tmp-dir" seLinuxMountContext=""
Oct 30 12:35:56.296667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296546    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8749c6a2-8a38-412c-9ea0-ed31e4a1f072" volumeName="kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-service-ca-bundle" seLinuxMountContext=""
Oct 30 12:35:56.296667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296558    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8749c6a2-8a38-412c-9ea0-ed31e4a1f072" volumeName="kubernetes.io/projected/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-kube-api-access-4pwxq" seLinuxMountContext=""
Oct 30 12:35:56.296667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296572    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a695456e-fedc-42fb-955b-5c55f356b0c3" volumeName="kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 12:35:56.296667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296583    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a695456e-fedc-42fb-955b-5c55f356b0c3" volumeName="kubernetes.io/projected/a695456e-fedc-42fb-955b-5c55f356b0c3-kube-api-access-vnhcl" seLinuxMountContext=""
Oct 30 12:35:56.296667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296594    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed" volumeName="kubernetes.io/secret/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed-cert" seLinuxMountContext=""
Oct 30 12:35:56.296667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296615    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d22f52c3-6029-4b19-aa6a-9843e569dbc4" volumeName="kubernetes.io/configmap/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-daemon-config" seLinuxMountContext=""
Oct 30 12:35:56.296667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296628    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7df7d7d2-f38d-4fef-8012-ef4a8c201277" volumeName="kubernetes.io/projected/7df7d7d2-f38d-4fef-8012-ef4a8c201277-kube-api-access-z8mjc" seLinuxMountContext=""
Oct 30 12:35:56.296667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296640    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e69cda5-58f4-409b-8575-d0d8f3c09ba6" volumeName="kubernetes.io/secret/8e69cda5-58f4-409b-8575-d0d8f3c09ba6-agent-certs" seLinuxMountContext=""
Oct 30 12:35:56.296667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296663    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd" volumeName="kubernetes.io/projected/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-kube-api-access-bnz5h" seLinuxMountContext=""
Oct 30 12:35:56.296667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296675    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b382eac-dd89-486d-b80b-1f5a402f23d4" volumeName="kubernetes.io/empty-dir/4b382eac-dd89-486d-b80b-1f5a402f23d4-tmp-dir" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296687    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c94f5d7-3c75-41bd-963f-a6fffa86747a" volumeName="kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296702    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a695456e-fedc-42fb-955b-5c55f356b0c3" volumeName="kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296716    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d22f52c3-6029-4b19-aa6a-9843e569dbc4" volumeName="kubernetes.io/configmap/d22f52c3-6029-4b19-aa6a-9843e569dbc4-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296730    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d22f52c3-6029-4b19-aa6a-9843e569dbc4" volumeName="kubernetes.io/projected/d22f52c3-6029-4b19-aa6a-9843e569dbc4-kube-api-access-prz95" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296758    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12bc2302-2423-4450-aeac-e30bd3c55ac9" volumeName="kubernetes.io/configmap/12bc2302-2423-4450-aeac-e30bd3c55ac9-signing-cabundle" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296769    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12bc2302-2423-4450-aeac-e30bd3c55ac9" volumeName="kubernetes.io/secret/12bc2302-2423-4450-aeac-e30bd3c55ac9-signing-key" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296780    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40905ce7-7c28-4d5c-a1a6-620f4d80dd5d" volumeName="kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovnkube-config" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296791    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b382eac-dd89-486d-b80b-1f5a402f23d4" volumeName="kubernetes.io/projected/4b382eac-dd89-486d-b80b-1f5a402f23d4-kube-api-access-dd6r5" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296803    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57060355-8a18-4a06-93bb-af1de546c2b7" volumeName="kubernetes.io/projected/57060355-8a18-4a06-93bb-af1de546c2b7-kube-api-access-655h6" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296814    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a695456e-fedc-42fb-955b-5c55f356b0c3" volumeName="kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296826    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2650cc31-aa41-466b-a61f-fd67f732ea03" volumeName="kubernetes.io/empty-dir/2650cc31-aa41-466b-a61f-fd67f732ea03-data-volume" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296839    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40905ce7-7c28-4d5c-a1a6-620f4d80dd5d" volumeName="kubernetes.io/projected/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-kube-api-access-cpvmw" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296851    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9a96919-4ab9-4bfe-9b8b-36d677c3d78c" volumeName="kubernetes.io/configmap/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-iptables-alerter-script" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296865    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12bc2302-2423-4450-aeac-e30bd3c55ac9" volumeName="kubernetes.io/projected/12bc2302-2423-4450-aeac-e30bd3c55ac9-kube-api-access-9xdt2" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296884    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7991e362-d106-4607-b8ee-ad70dfff9e7d" volumeName="kubernetes.io/empty-dir/7991e362-d106-4607-b8ee-ad70dfff9e7d-tmp" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296895    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8749c6a2-8a38-412c-9ea0-ed31e4a1f072" volumeName="kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296912    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8749c6a2-8a38-412c-9ea0-ed31e4a1f072" volumeName="kubernetes.io/empty-dir/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-archives-path" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296923    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e69cda5-58f4-409b-8575-d0d8f3c09ba6" volumeName="kubernetes.io/configmap/8e69cda5-58f4-409b-8575-d0d8f3c09ba6-konnectivity-ca" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296947    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3" volumeName="kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296959    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2650cc31-aa41-466b-a61f-fd67f732ea03" volumeName="kubernetes.io/projected/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-api-access-g6d6v" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296971    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40905ce7-7c28-4d5c-a1a6-620f4d80dd5d" volumeName="kubernetes.io/secret/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296983    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7991e362-d106-4607-b8ee-ad70dfff9e7d" volumeName="kubernetes.io/empty-dir/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-tuned" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.296995    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9a96919-4ab9-4bfe-9b8b-36d677c3d78c" volumeName="kubernetes.io/projected/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-kube-api-access-d5vcf" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297007    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed" volumeName="kubernetes.io/projected/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed-kube-api-access-rvh74" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297028    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e14fcd05-8067-4312-b9bb-90c4c9a15b23" volumeName="kubernetes.io/projected/e14fcd05-8067-4312-b9bb-90c4c9a15b23-kube-api-access-kg67k" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297039    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40905ce7-7c28-4d5c-a1a6-620f4d80dd5d" volumeName="kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297050    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40905ce7-7c28-4d5c-a1a6-620f4d80dd5d" volumeName="kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-env-overrides" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297061    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57060355-8a18-4a06-93bb-af1de546c2b7" volumeName="kubernetes.io/configmap/57060355-8a18-4a06-93bb-af1de546c2b7-config-volume" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297074    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7991e362-d106-4607-b8ee-ad70dfff9e7d" volumeName="kubernetes.io/projected/7991e362-d106-4607-b8ee-ad70dfff9e7d-kube-api-access-jsbfd" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297089    4522 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e14fcd05-8067-4312-b9bb-90c4c9a15b23" volumeName="kubernetes.io/configmap/e14fcd05-8067-4312-b9bb-90c4c9a15b23-serviceca" seLinuxMountContext=""
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297100    4522 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297108    4522 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297249    4522 factory.go:55] Registering systemd factory
Oct 30 12:35:56.297253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297269    4522 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:35:56.298723 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297569    4522 factory.go:153] Registering CRI-O factory
Oct 30 12:35:56.298723 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297582    4522 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:35:56.298723 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297644    4522 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:35:56.298723 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297668    4522 factory.go:103] Registering Raw factory
Oct 30 12:35:56.298723 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.297682    4522 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:35:56.298723 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.298040    4522 manager.go:319] Starting recovery of all containers
Oct 30 12:35:56.324692 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.324654    4522 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:35:56.326280 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.326254    4522 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:35:56.326280 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.326277    4522 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:35:56.326466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.326302    4522 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:35:56.326466 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:56.326345    4522 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 12:35:56.331258 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.331166    4522 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:56.336334 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.336310    4522 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:56.347085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.347057    4522 generic.go:356] "Generic (PLEG): container finished" podID="a695456e-fedc-42fb-955b-5c55f356b0c3" containerID="d4c4fde9d87a32db8f2978df239f925fd33b8f350ca08623e5fcbd4f0d29e6bc" exitCode=0
Oct 30 12:35:56.347085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.347084    4522 generic.go:356] "Generic (PLEG): container finished" podID="a695456e-fedc-42fb-955b-5c55f356b0c3" containerID="de3ac4bae78734c1144d37c04108e4c319fd268acef6608ddf4520b91bb9af2b" exitCode=0
Oct 30 12:35:56.347202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.347094    4522 generic.go:356] "Generic (PLEG): container finished" podID="a695456e-fedc-42fb-955b-5c55f356b0c3" containerID="9046885f3871eaa9712d2c06638f068ecdf50bc95c25f2575167d649150546b0" exitCode=0
Oct 30 12:35:56.347202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.347103    4522 generic.go:356] "Generic (PLEG): container finished" podID="a695456e-fedc-42fb-955b-5c55f356b0c3" containerID="256852000c5b7a52e7b880cb7158029c24bad61e29aa4269ce6988c33ec927a0" exitCode=0
Oct 30 12:35:56.347202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.347111    4522 generic.go:356] "Generic (PLEG): container finished" podID="a695456e-fedc-42fb-955b-5c55f356b0c3" containerID="21a96d86abdcd70e111a6f7d907478cd52280d12279720aef58a860303d8fa8e" exitCode=0
Oct 30 12:35:56.347202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.347119    4522 generic.go:356] "Generic (PLEG): container finished" podID="a695456e-fedc-42fb-955b-5c55f356b0c3" containerID="35cee61a413fa6eeb2d2484edb7f0e47749242e33a12eb3a12a7a90e53643eb0" exitCode=0
Oct 30 12:35:56.355313 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.355291    4522 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:56.355488 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.355314    4522 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:35:56.355551 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.355516    4522 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 12:35:56.356199 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.356172    4522 generic.go:356] "Generic (PLEG): container finished" podID="7ce8a1c7eb99ebcd1a86b5e56228777b" containerID="28ea0fb2a0328000c0e582a37a2d0cc561aa84b070f00921efa2fa920c8e370b" exitCode=0
Oct 30 12:35:56.365867 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.365853    4522 manager.go:324] Recovery completed
Oct 30 12:35:56.382639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.382622    4522 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:35:56.382639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.382636    4522 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:35:56.382748 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.382651    4522 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:35:56.382782 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.382765    4522 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 12:35:56.382815 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.382774    4522 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 12:35:56.382815 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.382789    4522 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 12:35:56.382815 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.382795    4522 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 12:35:56.382815 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.382800    4522 policy_none.go:50] "Start"
Oct 30 12:35:56.382815 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.382808    4522 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:35:56.382815 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.382816    4522 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:35:56.383436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.383422    4522 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 12:35:56.383472 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.383436    4522 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 12:35:56.383472 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.383443    4522 policy_none.go:44] "Start"
Oct 30 12:35:56.387181 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.387157    4522 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:35:56.389608 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.389594    4522 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:35:56.389678 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:56.389622    4522 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:35:56.389678 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.389634    4522 server.go:87] "Starting device plugin registration server"
Oct 30 12:35:56.389917 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.389903    4522 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:35:56.389989 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.389917    4522 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:35:56.390040 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.390019    4522 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:35:56.390161 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.390123    4522 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:35:56.390276 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.390169    4522 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:35:56.390276 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.390177    4522 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:35:56.390276 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.390211    4522 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:35:56.390155106Z","UUID":"a49aa042-d274-4a98-b0ba-0ff592ddeb04","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:35:56.391385 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:56.391366    4522 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:35:56.391514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.391496    4522 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:35:56.391594 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.391548    4522 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:35:56.426509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.426464    4522 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"]
Oct 30 12:35:56.426908 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.426882    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal" event={"ID":"3c78310b24bd820f4d32df7a2e462ac4","Type":"ContainerStarted","Data":"2d83ddc9065264a07479bb4e32edb16b0923d7e771df30d7b253d6f5c4f81a65"}
Oct 30 12:35:56.426962 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.426915    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal" event={"ID":"3c78310b24bd820f4d32df7a2e462ac4","Type":"ContainerStarted","Data":"37962d5f41d40956e0372be06ffe4fa326336ed3132a67a554e4855e51592964"}
Oct 30 12:35:56.427028 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.426959    4522 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.427124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.427109    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal" event={"ID":"7ce8a1c7eb99ebcd1a86b5e56228777b","Type":"ContainerStarted","Data":"a019a9fb7675a549ca3425934041846d4c4eb6d2f30a3ecf0690fee05f1f562e"}
Oct 30 12:35:56.427179 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.427133    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal" event={"ID":"7ce8a1c7eb99ebcd1a86b5e56228777b","Type":"ContainerDied","Data":"28ea0fb2a0328000c0e582a37a2d0cc561aa84b070f00921efa2fa920c8e370b"}
Oct 30 12:35:56.427179 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.427148    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal" event={"ID":"7ce8a1c7eb99ebcd1a86b5e56228777b","Type":"ContainerStarted","Data":"66cf76fc27b029e8af3e440792091cf93c3a7ad4ca65b17c66d224468cc05d7c"}
Oct 30 12:35:56.427179 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.427161    4522 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.447371 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.447330    4522 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:35:56.447458 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:56.447396    4522 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-133.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.447765 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.447752    4522 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:35:56.447805 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:56.447783    4522 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.490055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.490032    4522 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:35:56.491328 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.491312    4522 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:35:56.491446 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.491341    4522 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:35:56.491446 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.491369    4522 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-133.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:35:56.491446 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.491411    4522 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.529680 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.529662    4522 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.529748 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.529715    4522 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.556479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.556457    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3c78310b24bd820f4d32df7a2e462ac4-config\") pod \"kube-apiserver-proxy-ip-10-0-129-133.ec2.internal\" (UID: \"3c78310b24bd820f4d32df7a2e462ac4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.556584 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.556482    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ce8a1c7eb99ebcd1a86b5e56228777b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" (UID: \"7ce8a1c7eb99ebcd1a86b5e56228777b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.556584 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.556500    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ce8a1c7eb99ebcd1a86b5e56228777b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" (UID: \"7ce8a1c7eb99ebcd1a86b5e56228777b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.656806 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.656771    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3c78310b24bd820f4d32df7a2e462ac4-config\") pod \"kube-apiserver-proxy-ip-10-0-129-133.ec2.internal\" (UID: \"3c78310b24bd820f4d32df7a2e462ac4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.656806 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.656805    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ce8a1c7eb99ebcd1a86b5e56228777b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" (UID: \"7ce8a1c7eb99ebcd1a86b5e56228777b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.657031 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.656825    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ce8a1c7eb99ebcd1a86b5e56228777b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" (UID: \"7ce8a1c7eb99ebcd1a86b5e56228777b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.657031 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.656900    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3c78310b24bd820f4d32df7a2e462ac4-config\") pod \"kube-apiserver-proxy-ip-10-0-129-133.ec2.internal\" (UID: \"3c78310b24bd820f4d32df7a2e462ac4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.657031 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.656959    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ce8a1c7eb99ebcd1a86b5e56228777b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" (UID: \"7ce8a1c7eb99ebcd1a86b5e56228777b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:56.657031 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:56.656984    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ce8a1c7eb99ebcd1a86b5e56228777b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" (UID: \"7ce8a1c7eb99ebcd1a86b5e56228777b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:57.273184 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.273150    4522 apiserver.go:52] "Watching apiserver"
Oct 30 12:35:57.311837 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.311805    4522 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:35:57.312300 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.312275    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pfqxj","openshift-network-operator/iptables-alerter-9qbsn","kube-system/global-pull-secret-syncer-9lfhr","openshift-service-ca/service-ca-5fffc97bdf-xb6n4","openshift-multus/multus-additional-cni-plugins-6vxsv","openshift-ovn-kubernetes/ovnkube-node-l7v4x","openshift-cluster-node-tuning-operator/tuned-4sj5j","openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l","openshift-dns/dns-default-x5ck2","kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal","openshift-multus/multus-9r4kp","openshift-image-registry/node-ca-2s6fb","openshift-insights/periodic-gathering-x8599-trltf","openshift-multus/network-metrics-daemon-crnn8","kube-system/konnectivity-agent-j8d4p","openshift-dns/node-resolver-5n22h","openshift-insights/insights-runtime-extractor-p5vwm","openshift-network-diagnostics/network-check-target-rqz8x"]
Oct 30 12:35:57.312748 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.312728    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:57.316880 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.316861    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:57.320894 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.320873    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:57.326830 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.326809    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:35:57.326972 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.326955    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-54vc4\""
Oct 30 12:35:57.327216 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.327195    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:35:57.327312 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.327294    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:35:57.327397 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.327316    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:35:57.327397 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.327322    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rkrpr\""
Oct 30 12:35:57.327397 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.327348    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:35:57.339671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.339649    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tz7hb\""
Oct 30 12:35:57.339757 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.339705    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:35:57.339757 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.339722    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 12:35:57.339820 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.339799    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x6m2k\""
Oct 30 12:35:57.339820 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.339806    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:35:57.339876 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.339861    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kkflg\""
Oct 30 12:35:57.339994 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.339982    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6qrlt\""
Oct 30 12:35:57.340095 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340083    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 12:35:57.340141 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340132    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:35:57.340187 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340137    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gmdlh\""
Oct 30 12:35:57.340187 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340134    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2vwzh\""
Oct 30 12:35:57.340187 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340159    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-scxms\""
Oct 30 12:35:57.340187 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340164    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9zjv6\""
Oct 30 12:35:57.340187 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340179    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-562t6\""
Oct 30 12:35:57.340187 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340186    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qjz2q\""
Oct 30 12:35:57.340527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340423    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t86w7\""
Oct 30 12:35:57.340527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340428    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 12:35:57.340527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340459    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 12:35:57.340527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340468    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:35:57.340527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340479    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pdckv\""
Oct 30 12:35:57.340527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340504    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xkwtr\""
Oct 30 12:35:57.340707 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340549    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2mbkm\""
Oct 30 12:35:57.340707 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340606    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t6s7j\""
Oct 30 12:35:57.340707 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.340618    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bg2k7\""
Oct 30 12:35:57.365412 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.365393    4522 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:57.365542 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.365529    4522 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:57.387403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.387379    4522 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:35:57.387500 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:57.387425    4522 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-133.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:57.388148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.388135    4522 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:35:57.388210 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:57.388171    4522 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-133.ec2.internal"
Oct 30 12:35:57.396539 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.396524    4522 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:35:57.427294 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427273    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 12:35:57.427384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427321    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 12:35:57.427489 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427475    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:35:57.427557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427534    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:35:57.427557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427549    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:35:57.427666 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427559    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:35:57.427666 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427571    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 12:35:57.427666 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427583    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 12:35:57.427666 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427593    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 12:35:57.427666 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427606    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:35:57.427666 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427642    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:35:57.427666 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427668    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427714    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427726    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427803    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427810    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427810    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427827    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427845    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427846    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427863    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427863    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427876    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427898    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427911    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427941    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:35:57.427959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.427954    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:35:57.428486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.428022    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 12:35:57.428486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.428026    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:35:57.428486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.428040    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:35:57.428486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.428058    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:35:57.428486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.428072    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 12:35:57.428486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.428097    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:35:57.428486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.428416    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:35:57.433222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.433207    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 12:35:57.461367 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461335    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-socket-dir-parent\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.461440 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461377    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c94f5d7-3c75-41bd-963f-a6fffa86747a-kubelet-config\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:57.461440 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461402    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7991e362-d106-4607-b8ee-ad70dfff9e7d-tmp\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.461498 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461447    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-kubelet\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.461498 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461489    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-conf-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.461552 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461512    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7991e362-d106-4607-b8ee-ad70dfff9e7d-tmp\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.461552 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461525    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvh74\" (UniqueName: \"kubernetes.io/projected/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed-kube-api-access-rvh74\") pod \"ingress-canary-pfqxj\" (UID: \"bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed\") " pod="openshift-ingress-canary/ingress-canary-pfqxj"
Oct 30 12:35:57.461609 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461563    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-cnibin\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.461609 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461596    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhcl\" (UniqueName: \"kubernetes.io/projected/a695456e-fedc-42fb-955b-5c55f356b0c3-kube-api-access-vnhcl\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.461716 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461701    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c94f5d7-3c75-41bd-963f-a6fffa86747a-dbus\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:57.461813 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461795    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-sys-fs\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.461856 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461838    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-cni-multus\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.461902 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461885    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prz95\" (UniqueName: \"kubernetes.io/projected/d22f52c3-6029-4b19-aa6a-9843e569dbc4-kube-api-access-prz95\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.461935 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461925    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed-cert\") pod \"ingress-canary-pfqxj\" (UID: \"bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed\") " pod="openshift-ingress-canary/ingress-canary-pfqxj"
Oct 30 12:35:57.461972 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461947    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-etc-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.461972 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461963    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.462050 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461979    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovn-node-metrics-cert\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.462050 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.461998    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-var-lib-kubelet\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.462050 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462031    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-host\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.462179 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462110    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-cni-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.462179 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462134    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.462179 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462159    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.462332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462181    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2650cc31-aa41-466b-a61f-fd67f732ea03-data-volume\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:57.462332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462199    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-trusted-ca-bundle\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:57.462332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462215    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-kubernetes\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.462332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462265    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdt2\" (UniqueName: \"kubernetes.io/projected/12bc2302-2423-4450-aeac-e30bd3c55ac9-kube-api-access-9xdt2\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:57.462332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462266    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed-cert\") pod \"ingress-canary-pfqxj\" (UID: \"bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed\") " pod="openshift-ingress-canary/ingress-canary-pfqxj"
Oct 30 12:35:57.462332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462290    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-systemd-units\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.462332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462308    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovn-node-metrics-cert\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.462332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462314    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2650cc31-aa41-466b-a61f-fd67f732ea03-data-volume\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:57.462719 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462407    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-systemd\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.462719 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462435    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-cni-netd\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.462719 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462501    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-archives-path\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:57.462719 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462553    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e14fcd05-8067-4312-b9bb-90c4c9a15b23-serviceca\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:57.462719 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462588    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-archives-path\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:57.462719 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462595    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-modprobe-d\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.462719 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462653    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.462719 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462692    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysctl-d\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.463085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462735    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vcf\" (UniqueName: \"kubernetes.io/projected/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-kube-api-access-d5vcf\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:57.463085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462788    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/12bc2302-2423-4450-aeac-e30bd3c55ac9-signing-cabundle\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:57.463085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462791    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e14fcd05-8067-4312-b9bb-90c4c9a15b23-serviceca\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:57.463085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462821    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4b382eac-dd89-486d-b80b-1f5a402f23d4-hosts-file\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:57.463085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462930    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-kubelet\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.463085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.462988    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-sys\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.463085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463005    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/12bc2302-2423-4450-aeac-e30bd3c55ac9-signing-cabundle\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:57.463304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463097    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-trusted-ca-bundle\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:57.463304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463120    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.463304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463140    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-netns\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.463304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463161    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-multus-certs\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.463304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463201    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pwxq\" (UniqueName: \"kubernetes.io/projected/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-kube-api-access-4pwxq\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:57.463304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463233    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.463496 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463338    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:57.463496 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463378    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e69cda5-58f4-409b-8575-d0d8f3c09ba6-agent-certs\") pod \"konnectivity-agent-j8d4p\" (UID: \"8e69cda5-58f4-409b-8575-d0d8f3c09ba6\") " pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:57.463496 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463418    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-run-ovn-kubernetes\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.463631 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463457    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-cni-bin\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.464049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463793    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-tuned\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.464049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463879    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6r5\" (UniqueName: \"kubernetes.io/projected/4b382eac-dd89-486d-b80b-1f5a402f23d4-kube-api-access-dd6r5\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:57.464049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463922    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.464049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463986    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-tuned\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.464049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464009    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e69cda5-58f4-409b-8575-d0d8f3c09ba6-agent-certs\") pod \"konnectivity-agent-j8d4p\" (UID: \"8e69cda5-58f4-409b-8575-d0d8f3c09ba6\") " pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:57.464049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.463988    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.464366 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464068    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-socket-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.464366 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464097    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-daemon-config\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.464366 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464168    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnz5h\" (UniqueName: \"kubernetes.io/projected/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-kube-api-access-bnz5h\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:57.464366 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464234    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-system-cni-dir\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.464366 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464270    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b382eac-dd89-486d-b80b-1f5a402f23d4-tmp-dir\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:57.464366 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464303    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovnkube-script-lib\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.464366 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464332    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e14fcd05-8067-4312-b9bb-90c4c9a15b23-host\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:57.464671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464418    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsbfd\" (UniqueName: \"kubernetes.io/projected/7991e362-d106-4607-b8ee-ad70dfff9e7d-kube-api-access-jsbfd\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.464671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464453    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8s2g\" (UniqueName: \"kubernetes.io/projected/02a59e47-332d-40bb-af4a-2a0b59a816c8-kube-api-access-h8s2g\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.464671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464485    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-k8s-cni-cncf-io\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.464671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464514    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-host-slash\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:57.464671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464550    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-ovn\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.464671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464576    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvmw\" (UniqueName: \"kubernetes.io/projected/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-kube-api-access-cpvmw\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.464671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464606    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:57.464671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464635    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-cnibin\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.464671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464666    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-cni-bin\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.465023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464695    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-etc-kubernetes\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.465023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464720    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e69cda5-58f4-409b-8575-d0d8f3c09ba6-konnectivity-ca\") pod \"konnectivity-agent-j8d4p\" (UID: \"8e69cda5-58f4-409b-8575-d0d8f3c09ba6\") " pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:57.465023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464752    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655h6\" (UniqueName: \"kubernetes.io/projected/57060355-8a18-4a06-93bb-af1de546c2b7-kube-api-access-655h6\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:57.465023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464784    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8mjc\" (UniqueName: \"kubernetes.io/projected/7df7d7d2-f38d-4fef-8012-ef4a8c201277-kube-api-access-z8mjc\") pod \"migrator-66df5d496b-24kzf\" (UID: \"7df7d7d2-f38d-4fef-8012-ef4a8c201277\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf"
Oct 30 12:35:57.465023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464814    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-run-netns\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.465023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464842    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-node-log\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.465023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464867    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-device-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.465023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464897    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d22f52c3-6029-4b19-aa6a-9843e569dbc4-cni-binary-copy\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.465023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464925    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:57.465023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464953    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-log-socket\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.465023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.464983    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovnkube-config\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.465023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465008    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-run\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465071    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-lib-modules\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465100    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465129    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-hostroot\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465161    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465192    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57060355-8a18-4a06-93bb-af1de546c2b7-tmp-dir\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465217    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-var-lib-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465248    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-env-overrides\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465315    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysctl-conf\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465369    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465404    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2650cc31-aa41-466b-a61f-fd67f732ea03-crio-socket\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465437    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-slash\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465467    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg67k\" (UniqueName: \"kubernetes.io/projected/e14fcd05-8067-4312-b9bb-90c4c9a15b23-kube-api-access-kg67k\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465497    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysconfig\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.465528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465525    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-systemd\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.466099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465558    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-iptables-alerter-script\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:57.466099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465594    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:57.466099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465625    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/12bc2302-2423-4450-aeac-e30bd3c55ac9-signing-key\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:57.466099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465657    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6d6v\" (UniqueName: \"kubernetes.io/projected/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-api-access-g6d6v\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:57.466099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465687    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57060355-8a18-4a06-93bb-af1de546c2b7-config-volume\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:57.466099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465714    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m442\" (UniqueName: \"kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442\") pod \"network-check-target-rqz8x\" (UID: \"b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3\") " pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:57.466099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465746    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-service-ca-bundle\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:57.466099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465777    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-registration-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.466099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.465805    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-system-cni-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.466489 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.466142    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-daemon-config\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.466535 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.466510    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b382eac-dd89-486d-b80b-1f5a402f23d4-tmp-dir\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:57.466779 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.466761    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovnkube-script-lib\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.467053 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.467035    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e69cda5-58f4-409b-8575-d0d8f3c09ba6-konnectivity-ca\") pod \"konnectivity-agent-j8d4p\" (UID: \"8e69cda5-58f4-409b-8575-d0d8f3c09ba6\") " pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:35:57.467121 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.467082    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57060355-8a18-4a06-93bb-af1de546c2b7-tmp-dir\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:57.467420 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.467394    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-env-overrides\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.467559 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.467534    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-iptables-alerter-script\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:57.467559 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.467546    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/12bc2302-2423-4450-aeac-e30bd3c55ac9-signing-key\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:57.467735 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.467710    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-service-ca-bundle\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:57.467983 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.467798    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c94f5d7-3c75-41bd-963f-a6fffa86747a-original-pull-secret\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:57.467983 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.467869    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-ovnkube-config\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.467983 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.467928    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d22f52c3-6029-4b19-aa6a-9843e569dbc4-cni-binary-copy\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.468160 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.467987    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a695456e-fedc-42fb-955b-5c55f356b0c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.468543 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.468526    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57060355-8a18-4a06-93bb-af1de546c2b7-config-volume\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:57.479220 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.479198    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhcl\" (UniqueName: \"kubernetes.io/projected/a695456e-fedc-42fb-955b-5c55f356b0c3-kube-api-access-vnhcl\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.479297 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.479271    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prz95\" (UniqueName: \"kubernetes.io/projected/d22f52c3-6029-4b19-aa6a-9843e569dbc4-kube-api-access-prz95\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.479409 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.479392    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvh74\" (UniqueName: \"kubernetes.io/projected/bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed-kube-api-access-rvh74\") pod \"ingress-canary-pfqxj\" (UID: \"bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed\") " pod="openshift-ingress-canary/ingress-canary-pfqxj"
Oct 30 12:35:57.479444 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.479429    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdt2\" (UniqueName: \"kubernetes.io/projected/12bc2302-2423-4450-aeac-e30bd3c55ac9-kube-api-access-9xdt2\") pod \"service-ca-5fffc97bdf-xb6n4\" (UID: \"12bc2302-2423-4450-aeac-e30bd3c55ac9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4"
Oct 30 12:35:57.479619 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.479603    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vcf\" (UniqueName: \"kubernetes.io/projected/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-kube-api-access-d5vcf\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:57.480420 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.480406    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6r5\" (UniqueName: \"kubernetes.io/projected/4b382eac-dd89-486d-b80b-1f5a402f23d4-kube-api-access-dd6r5\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:57.480786 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.480772    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pwxq\" (UniqueName: \"kubernetes.io/projected/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-kube-api-access-4pwxq\") pod \"periodic-gathering-x8599-trltf\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") " pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:35:57.482540 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.482521    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnz5h\" (UniqueName: \"kubernetes.io/projected/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-kube-api-access-bnz5h\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:57.483720 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.483700    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsbfd\" (UniqueName: \"kubernetes.io/projected/7991e362-d106-4607-b8ee-ad70dfff9e7d-kube-api-access-jsbfd\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.483897 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.483880    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8mjc\" (UniqueName: \"kubernetes.io/projected/7df7d7d2-f38d-4fef-8012-ef4a8c201277-kube-api-access-z8mjc\") pod \"migrator-66df5d496b-24kzf\" (UID: \"7df7d7d2-f38d-4fef-8012-ef4a8c201277\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24kzf"
Oct 30 12:35:57.484006 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.483985    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-655h6\" (UniqueName: \"kubernetes.io/projected/57060355-8a18-4a06-93bb-af1de546c2b7-kube-api-access-655h6\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:57.484095 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.484081    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg67k\" (UniqueName: \"kubernetes.io/projected/e14fcd05-8067-4312-b9bb-90c4c9a15b23-kube-api-access-kg67k\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:57.484256 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.484238    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m442\" (UniqueName: \"kubernetes.io/projected/b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3-kube-api-access-2m442\") pod \"network-check-target-rqz8x\" (UID: \"b9cc4ae3-3f1f-4bb5-9dfb-765f25583cb3\") " pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:57.484448 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.484436    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvmw\" (UniqueName: \"kubernetes.io/projected/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-kube-api-access-cpvmw\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.484777 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.484762    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6d6v\" (UniqueName: \"kubernetes.io/projected/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-api-access-g6d6v\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:57.484825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.484810    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8s2g\" (UniqueName: \"kubernetes.io/projected/02a59e47-332d-40bb-af4a-2a0b59a816c8-kube-api-access-h8s2g\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.562971 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.562927    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5n22h_4b382eac-dd89-486d-b80b-1f5a402f23d4/dns-node-resolver/0.log"
Oct 30 12:35:57.566057 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566038    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-k8s-cni-cncf-io\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.566147 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566062    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-host-slash\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:57.566147 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566077    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-ovn\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.566147 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566093    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-cnibin\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.566147 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566133    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9a96919-4ab9-4bfe-9b8b-36d677c3d78c-host-slash\") pod \"iptables-alerter-9qbsn\" (UID: \"a9a96919-4ab9-4bfe-9b8b-36d677c3d78c\") " pod="openshift-network-operator/iptables-alerter-9qbsn"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566145    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-cnibin\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566142    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-ovn\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566158    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-cni-bin\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566169    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-k8s-cni-cncf-io\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566179    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-cni-bin\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566188    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-etc-kubernetes\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566209    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-run-netns\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566223    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-node-log\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566238    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-device-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566244    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-node-log\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566253    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566223    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-etc-kubernetes\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566238    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-run-netns\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566271    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-device-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566268    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-log-socket\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566294    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-log-socket\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566301    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-run\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566321    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-lib-modules\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.566332 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566336    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566365    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-hostroot\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566374    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-run\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566384    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:57.566348    4522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566407    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-var-lib-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566419    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566419    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-hostroot\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566429    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysctl-conf\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566447    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-var-lib-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:57.566467    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls podName:57060355-8a18-4a06-93bb-af1de546c2b7 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:58.066447628 +0000 UTC m=+1.881774116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls") pod "dns-default-x5ck2" (UID: "57060355-8a18-4a06-93bb-af1de546c2b7") : secret "dns-default-metrics-tls" not found
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566465    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-lib-modules\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:57.566477    4522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566487    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2650cc31-aa41-466b-a61f-fd67f732ea03-crio-socket\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566513    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-slash\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566517    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2650cc31-aa41-466b-a61f-fd67f732ea03-crio-socket\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:57.566530    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs podName:4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd nodeName:}" failed. No retries permitted until 2026-10-30 12:35:58.066514895 +0000 UTC m=+1.881841367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs") pod "network-metrics-daemon-crnn8" (UID: "4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd") : secret "metrics-daemon-secret" not found
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566546    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-slash\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566548    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysctl-conf\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566572    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysconfig\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566597    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-systemd\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566617    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566646    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-registration-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566648    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysconfig\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566657    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-systemd\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566692    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-registration-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566723    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-system-cni-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566755    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-socket-dir-parent\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566780    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c94f5d7-3c75-41bd-963f-a6fffa86747a-kubelet-config\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566805    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-kubelet\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566815    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-system-cni-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566821    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-socket-dir-parent\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566827    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-conf-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566853    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c94f5d7-3c75-41bd-963f-a6fffa86747a-kubelet-config\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566864    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-kubelet\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566874    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-conf-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566889    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-cnibin\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566866    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-cnibin\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566912    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c94f5d7-3c75-41bd-963f-a6fffa86747a-dbus\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566928    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-sys-fs\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566953    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-cni-multus\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566972    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c94f5d7-3c75-41bd-963f-a6fffa86747a-dbus\") pod \"global-pull-secret-syncer-9lfhr\" (UID: \"8c94f5d7-3c75-41bd-963f-a6fffa86747a\") " pod="kube-system/global-pull-secret-syncer-9lfhr"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566978    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-etc-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.566988    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-sys-fs\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567014    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-var-lib-cni-multus\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567029    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-etc-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567033    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567048    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-openvswitch\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.567055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567066    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-var-lib-kubelet\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567090    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-host\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567095    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-var-lib-kubelet\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567111    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-cni-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567127    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-host\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567135    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567138    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2650cc31-aa41-466b-a61f-fd67f732ea03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567159    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-kubernetes\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567166    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-multus-cni-dir\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567183    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-systemd-units\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567195    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-kubernetes\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567197    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567207    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-systemd\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567217    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-systemd-units\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567227    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-cni-netd\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567243    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-modprobe-d\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567255    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-run-systemd\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567256    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysctl-d\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567278    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4b382eac-dd89-486d-b80b-1f5a402f23d4-hosts-file\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567306    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-cni-netd\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567314    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4b382eac-dd89-486d-b80b-1f5a402f23d4-hosts-file\") pod \"node-resolver-5n22h\" (UID: \"4b382eac-dd89-486d-b80b-1f5a402f23d4\") " pod="openshift-dns/node-resolver-5n22h"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567330    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-kubelet\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567367    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-modprobe-d\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567371    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-sys\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567371    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-etc-sysctl-d\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567388    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-kubelet\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567395    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567407    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7991e362-d106-4607-b8ee-ad70dfff9e7d-sys\") pod \"tuned-4sj5j\" (UID: \"7991e362-d106-4607-b8ee-ad70dfff9e7d\") " pod="openshift-cluster-node-tuning-operator/tuned-4sj5j"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567419    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-netns\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567424    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567439    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-multus-certs\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567457    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567475    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-run-ovn-kubernetes\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567474    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-netns\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567496    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-cni-bin\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567527    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d22f52c3-6029-4b19-aa6a-9843e569dbc4-host-run-multus-certs\") pod \"multus-9r4kp\" (UID: \"d22f52c3-6029-4b19-aa6a-9843e569dbc4\") " pod="openshift-multus/multus-9r4kp"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:57.567537    4522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567528    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-run-ovn-kubernetes\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567542    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-cni-bin\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567542    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567560    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40905ce7-7c28-4d5c-a1a6-620f4d80dd5d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l7v4x\" (UID: \"40905ce7-7c28-4d5c-a1a6-620f4d80dd5d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567569    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-socket-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:57.567579    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls podName:2650cc31-aa41-466b-a61f-fd67f732ea03 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:58.067568785 +0000 UTC m=+1.882895277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p5vwm" (UID: "2650cc31-aa41-466b-a61f-fd67f732ea03") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567601    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02a59e47-332d-40bb-af4a-2a0b59a816c8-socket-dir\") pod \"aws-ebs-csi-driver-node-r287l\" (UID: \"02a59e47-332d-40bb-af4a-2a0b59a816c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r287l"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567604    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-system-cni-dir\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567619    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e14fcd05-8067-4312-b9bb-90c4c9a15b23-host\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567658    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e14fcd05-8067-4312-b9bb-90c4c9a15b23-host\") pod \"node-ca-2s6fb\" (UID: \"e14fcd05-8067-4312-b9bb-90c4c9a15b23\") " pod="openshift-image-registry/node-ca-2s6fb"
Oct 30 12:35:57.568374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.567659    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a695456e-fedc-42fb-955b-5c55f356b0c3-system-cni-dir\") pod \"multus-additional-cni-plugins-6vxsv\" (UID: \"a695456e-fedc-42fb-955b-5c55f356b0c3\") " pod="openshift-multus/multus-additional-cni-plugins-6vxsv"
Oct 30 12:35:57.622336 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:57.622321    4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:35:58.022186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:58.022165    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:58.072167 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:58.072139    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:58.072256 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:58.072176    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:58.072256 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:58.072196    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:58.072334 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:58.072273    4522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:58.072334 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:58.072290    4522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:35:58.072334 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:58.072300    4522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:35:58.072334 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:58.072328    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls podName:2650cc31-aa41-466b-a61f-fd67f732ea03 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:59.072316567 +0000 UTC m=+2.887643046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p5vwm" (UID: "2650cc31-aa41-466b-a61f-fd67f732ea03") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:58.072473 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:58.072373    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs podName:4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd nodeName:}" failed. No retries permitted until 2026-10-30 12:35:59.072335752 +0000 UTC m=+2.887662233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs") pod "network-metrics-daemon-crnn8" (UID: "4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd") : secret "metrics-daemon-secret" not found
Oct 30 12:35:58.072473 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:58.072392    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls podName:57060355-8a18-4a06-93bb-af1de546c2b7 nodeName:}" failed. No retries permitted until 2026-10-30 12:35:59.072383042 +0000 UTC m=+2.887709524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls") pod "dns-default-x5ck2" (UID: "57060355-8a18-4a06-93bb-af1de546c2b7") : secret "dns-default-metrics-tls" not found
Oct 30 12:35:58.122952 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:58.122933    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:58.258448 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:58.258425    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:35:58.369396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:58.369322    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xb6n4" event={"ID":"12bc2302-2423-4450-aeac-e30bd3c55ac9","Type":"ContainerStarted","Data":"7595d3055e87bf61d21cb318591b4748d8e44353f3edd2f6610623887c4dfdf1"}
Oct 30 12:35:58.564012 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:58.563986    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2s6fb_e14fcd05-8067-4312-b9bb-90c4c9a15b23/node-ca/0.log"
Oct 30 12:35:58.889686 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:58.889638    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:58.990683 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:58.990637    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:59.075566 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:59.075527    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:59.083532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:59.083504    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:35:59.083644 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:59.083543    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:35:59.083644 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:59.083564    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:35:59.083718 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:59.083652    4522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:35:59.083718 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:59.083695    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs podName:4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd nodeName:}" failed. No retries permitted until 2026-10-30 12:36:01.0836846 +0000 UTC m=+4.899011068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs") pod "network-metrics-daemon-crnn8" (UID: "4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd") : secret "metrics-daemon-secret" not found
Oct 30 12:35:59.083795 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:59.083653    4522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:35:59.083795 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:59.083762    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls podName:57060355-8a18-4a06-93bb-af1de546c2b7 nodeName:}" failed. No retries permitted until 2026-10-30 12:36:01.083750667 +0000 UTC m=+4.899077136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls") pod "dns-default-x5ck2" (UID: "57060355-8a18-4a06-93bb-af1de546c2b7") : secret "dns-default-metrics-tls" not found
Oct 30 12:35:59.083795 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:59.083653    4522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:59.083795 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:35:59.083792    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls podName:2650cc31-aa41-466b-a61f-fd67f732ea03 nodeName:}" failed. No retries permitted until 2026-10-30 12:36:01.083786752 +0000 UTC m=+4.899113218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p5vwm" (UID: "2650cc31-aa41-466b-a61f-fd67f732ea03") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:35:59.364063 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:59.364033    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pfqxj_bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed/serve-healthcheck-canary/0.log"
Oct 30 12:35:59.425987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:59.425960    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:59.746880 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:59.746808    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:35:59.763334 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:35:59.763313    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-24kzf_7df7d7d2-f38d-4fef-8012-ef4a8c201277/migrator/0.log"
Oct 30 12:36:00.133860 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:00.133831    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:00.331116 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:00.331075    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:00.534905 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:00.534877    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:00.722865 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:00.722827    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:00.874669 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:00.874600    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:36:00.924668 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:00.924640    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:01.096282 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:01.096243    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:36:01.096478 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:01.096299    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:36:01.096478 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:01.096369    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:36:01.096478 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:36:01.096427    4522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:36:01.096478 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:36:01.096454    4522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:36:01.096630 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:36:01.096490    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls podName:57060355-8a18-4a06-93bb-af1de546c2b7 nodeName:}" failed. No retries permitted until 2026-10-30 12:36:05.096473437 +0000 UTC m=+8.911799924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls") pod "dns-default-x5ck2" (UID: "57060355-8a18-4a06-93bb-af1de546c2b7") : secret "dns-default-metrics-tls" not found
Oct 30 12:36:01.096630 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:36:01.096509    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs podName:4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd nodeName:}" failed. No retries permitted until 2026-10-30 12:36:05.09650016 +0000 UTC m=+8.911826634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs") pod "network-metrics-daemon-crnn8" (UID: "4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd") : secret "metrics-daemon-secret" not found
Oct 30 12:36:01.096784 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:01.096767    4522 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:36:01.098863 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:01.098830    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2650cc31-aa41-466b-a61f-fd67f732ea03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p5vwm\" (UID: \"2650cc31-aa41-466b-a61f-fd67f732ea03\") " pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:36:01.185874 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:01.185793    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:01.222086 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:01.222060    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p5vwm"
Oct 30 12:36:01.521369 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:01.521335    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p5vwm"]
Oct 30 12:36:01.523702 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:36:01.523674    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2650cc31_aa41_466b_a61f_fd67f732ea03.slice/crio-f7a2419f6c1f594e7991cedc66cfdc64b66c68cce51263aa7708f63f8f81181e WatchSource:0}: Error finding container f7a2419f6c1f594e7991cedc66cfdc64b66c68cce51263aa7708f63f8f81181e: Status 404 returned error can't find the container with id f7a2419f6c1f594e7991cedc66cfdc64b66c68cce51263aa7708f63f8f81181e
Oct 30 12:36:02.166782 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:02.166740    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:02.321839 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:02.321813    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:02.381464 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:02.381439    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p5vwm" event={"ID":"2650cc31-aa41-466b-a61f-fd67f732ea03","Type":"ContainerStarted","Data":"253851ab22e903b999481eae1ddc27b7a8d858224790fd4cf2fcd7bc7c3c740d"}
Oct 30 12:36:02.381586 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:02.381519    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p5vwm" event={"ID":"2650cc31-aa41-466b-a61f-fd67f732ea03","Type":"ContainerStarted","Data":"94868fe97156bb53fd604d7efaa3e7fd075c25c759742aafb1ccf7bb8e2c5dc3"}
Oct 30 12:36:02.381586 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:02.381540    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p5vwm" event={"ID":"2650cc31-aa41-466b-a61f-fd67f732ea03","Type":"ContainerStarted","Data":"f7a2419f6c1f594e7991cedc66cfdc64b66c68cce51263aa7708f63f8f81181e"}
Oct 30 12:36:02.521098 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:02.521070    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:02.727213 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:02.727178    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:02.919410 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:02.919312    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:03.378420 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:03.378387    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:03.464616 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:03.464591    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:36:03.464802 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:03.464789    4522 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:36:03.480416 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:03.480386    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:03.589430 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:03.589370    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:03.793798 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:03.793763    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:04.129620 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:04.129522    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:04.332183 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:04.332151    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:04.389806 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:04.389718    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p5vwm" event={"ID":"2650cc31-aa41-466b-a61f-fd67f732ea03","Type":"ContainerStarted","Data":"7790b6c435df735614e0b481dfc34a873015c6bdbc345c33a16a774f8f1b143d"}
Oct 30 12:36:04.524514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:04.524481    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:04.722019 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:04.721932    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:04.927751 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:04.927721    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:05.130080 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:05.130037    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:36:05.130272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:05.130089    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:36:05.130272 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:36:05.130182    4522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:36:05.130272 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:36:05.130249    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls podName:57060355-8a18-4a06-93bb-af1de546c2b7 nodeName:}" failed. No retries permitted until 2026-10-30 12:36:13.130233788 +0000 UTC m=+16.945560273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls") pod "dns-default-x5ck2" (UID: "57060355-8a18-4a06-93bb-af1de546c2b7") : secret "dns-default-metrics-tls" not found
Oct 30 12:36:05.132214 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:05.132187    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd-metrics-certs\") pod \"network-metrics-daemon-crnn8\" (UID: \"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd\") " pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:36:05.134726 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:05.134706    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l7v4x"
Oct 30 12:36:05.413422 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:05.413325    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crnn8"
Oct 30 12:36:05.702880 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:05.702843    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-crnn8"]
Oct 30 12:36:05.704339 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:36:05.704308    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0b65cc_3c7f_4ada_ad05_7a82aaa7a9dd.slice/crio-3b8c790d3e06e71a6ed361dea2918d4867a50531e4c2145dc92f6681411e2336 WatchSource:0}: Error finding container 3b8c790d3e06e71a6ed361dea2918d4867a50531e4c2145dc92f6681411e2336: Status 404 returned error can't find the container with id 3b8c790d3e06e71a6ed361dea2918d4867a50531e4c2145dc92f6681411e2336
Oct 30 12:36:06.126407 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:06.126368    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:06.227589 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:06.227544    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:06.330080 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:06.330048    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:06.396510 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:06.396427    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-crnn8" event={"ID":"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd","Type":"ContainerStarted","Data":"3b8c790d3e06e71a6ed361dea2918d4867a50531e4c2145dc92f6681411e2336"}
Oct 30 12:36:06.525099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:06.525032    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:06.728789 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:06.728754    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:06.834649 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:06.834620    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:06.936132 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:06.936057    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:07.401187 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:07.401150    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-crnn8" event={"ID":"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd","Type":"ContainerStarted","Data":"af7478126ac21207abb536592bb494764a5a2d8f37fa48486550234ca8aa6953"}
Oct 30 12:36:07.401421 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:07.401232    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-crnn8" event={"ID":"4b0b65cc-3c7f-4ada-ad05-7a82aaa7a9dd","Type":"ContainerStarted","Data":"9b9376ba3a2102e5bdc3838733d1de26c8c77f02eb898744c7b6d72f0483d15e"}
Oct 30 12:36:08.134060 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:08.134034    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:08.274611 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:08.274584    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:08.373824 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:08.373790    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:08.476414 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:08.476326    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:08.574299 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:08.574259    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:08.722864 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:08.722827    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:08.922376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:08.922321    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:09.726279 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:09.726245    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:09.939338 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:09.939306    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:10.137049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:10.137015    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:10.292370 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:10.292322    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:10.392799 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:10.392725    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:13.018239 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:13.018204    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:13.118648 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:13.118617    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:13.193210 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:13.193177    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:36:13.195578 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:13.195547    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57060355-8a18-4a06-93bb-af1de546c2b7-metrics-tls\") pod \"dns-default-x5ck2\" (UID: \"57060355-8a18-4a06-93bb-af1de546c2b7\") " pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:36:13.218486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:13.218447    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:36:13.508756 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:13.508729    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x5ck2"]
Oct 30 12:36:13.511581 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:36:13.511558    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57060355_8a18_4a06_93bb_af1de546c2b7.slice/crio-be4855625f37d9ba240afdac200022350329884d638aa645dcb0197cb990f880 WatchSource:0}: Error finding container be4855625f37d9ba240afdac200022350329884d638aa645dcb0197cb990f880: Status 404 returned error can't find the container with id be4855625f37d9ba240afdac200022350329884d638aa645dcb0197cb990f880
Oct 30 12:36:14.425123 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:14.425087    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x5ck2" event={"ID":"57060355-8a18-4a06-93bb-af1de546c2b7","Type":"ContainerStarted","Data":"be4855625f37d9ba240afdac200022350329884d638aa645dcb0197cb990f880"}
Oct 30 12:36:15.322064 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.322033    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hjc4m"]
Oct 30 12:36:15.325084 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.325064    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hjc4m"
Oct 30 12:36:15.333979 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.333947    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-f67bl"]
Oct 30 12:36:15.336853 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.336833    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-f67bl"
Oct 30 12:36:15.345436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.345407    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hjc4m"]
Oct 30 12:36:15.345554 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.345519    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 12:36:15.345640 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.345622    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-h4l8k\""
Oct 30 12:36:15.356632 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.356612    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 12:36:15.356978 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.356950    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 12:36:15.357317 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.357295    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-f67bl"]
Oct 30 12:36:15.408152 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.408114    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e93f8976-4436-4f2f-afd7-adc79bd9e975-tmp\") pod \"downloads-6dd789c8b7-f67bl\" (UID: \"e93f8976-4436-4f2f-afd7-adc79bd9e975\") " pod="openshift-console/downloads-6dd789c8b7-f67bl"
Oct 30 12:36:15.408346 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.408159    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2jbm\" (UniqueName: \"kubernetes.io/projected/e93f8976-4436-4f2f-afd7-adc79bd9e975-kube-api-access-h2jbm\") pod \"downloads-6dd789c8b7-f67bl\" (UID: \"e93f8976-4436-4f2f-afd7-adc79bd9e975\") " pod="openshift-console/downloads-6dd789c8b7-f67bl"
Oct 30 12:36:15.408431 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.408377    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d0cd6d0a-9b96-449a-846f-c57eb11d02e2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hjc4m\" (UID: \"d0cd6d0a-9b96-449a-846f-c57eb11d02e2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hjc4m"
Oct 30 12:36:15.417432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.417400    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:15.422445 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.422421    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-674fd5846d-bn6zd"]
Oct 30 12:36:15.425288 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.425272    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.434087 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.434047    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x5ck2" event={"ID":"57060355-8a18-4a06-93bb-af1de546c2b7","Type":"ContainerStarted","Data":"c8ecdebf3084fec10426684422c1fe549474a8992fda147b4d479af53856fdec"}
Oct 30 12:36:15.434222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.434133    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x5ck2" event={"ID":"57060355-8a18-4a06-93bb-af1de546c2b7","Type":"ContainerStarted","Data":"aa9821cdcc6788474ef701564cb914c905a35001d456e1924f569f61de0d56b9"}
Oct 30 12:36:15.434222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.434192    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:36:15.456214 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.456179    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-c5bfj\""
Oct 30 12:36:15.456403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.456282    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 12:36:15.456403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.456312    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 12:36:15.456403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.456345    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 12:36:15.456403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.456347    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 12:36:15.456403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.456391    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vtqg8\""
Oct 30 12:36:15.460787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.460766    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-674fd5846d-bn6zd"]
Oct 30 12:36:15.508905 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.508860    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e93f8976-4436-4f2f-afd7-adc79bd9e975-tmp\") pod \"downloads-6dd789c8b7-f67bl\" (UID: \"e93f8976-4436-4f2f-afd7-adc79bd9e975\") " pod="openshift-console/downloads-6dd789c8b7-f67bl"
Oct 30 12:36:15.509104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.508911    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-installation-pull-secrets\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.509104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.508942    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2jbm\" (UniqueName: \"kubernetes.io/projected/e93f8976-4436-4f2f-afd7-adc79bd9e975-kube-api-access-h2jbm\") pod \"downloads-6dd789c8b7-f67bl\" (UID: \"e93f8976-4436-4f2f-afd7-adc79bd9e975\") " pod="openshift-console/downloads-6dd789c8b7-f67bl"
Oct 30 12:36:15.509104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.508969    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d0cd6d0a-9b96-449a-846f-c57eb11d02e2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hjc4m\" (UID: \"d0cd6d0a-9b96-449a-846f-c57eb11d02e2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hjc4m"
Oct 30 12:36:15.509104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.508995    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-image-registry-private-configuration\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.509104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.509033    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-registry-tls\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.509104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.509070    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-registry-certificates\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.509566 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.509211    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-926zw\" (UniqueName: \"kubernetes.io/projected/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-kube-api-access-926zw\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.509566 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.509236    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-trusted-ca\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.509566 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.509252    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-bound-sa-token\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.509566 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.509270    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-ca-trust-extracted\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.509566 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.509310    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e93f8976-4436-4f2f-afd7-adc79bd9e975-tmp\") pod \"downloads-6dd789c8b7-f67bl\" (UID: \"e93f8976-4436-4f2f-afd7-adc79bd9e975\") " pod="openshift-console/downloads-6dd789c8b7-f67bl"
Oct 30 12:36:15.511556 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.511535    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d0cd6d0a-9b96-449a-846f-c57eb11d02e2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hjc4m\" (UID: \"d0cd6d0a-9b96-449a-846f-c57eb11d02e2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hjc4m"
Oct 30 12:36:15.518604 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.518575    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:15.527574 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.527550    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2jbm\" (UniqueName: \"kubernetes.io/projected/e93f8976-4436-4f2f-afd7-adc79bd9e975-kube-api-access-h2jbm\") pod \"downloads-6dd789c8b7-f67bl\" (UID: \"e93f8976-4436-4f2f-afd7-adc79bd9e975\") " pod="openshift-console/downloads-6dd789c8b7-f67bl"
Oct 30 12:36:15.609621 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.609539    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-installation-pull-secrets\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.609621 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.609577    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-image-registry-private-configuration\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.609621 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.609595    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-registry-tls\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.609621 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.609617    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-registry-certificates\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.609977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.609648    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-926zw\" (UniqueName: \"kubernetes.io/projected/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-kube-api-access-926zw\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.609977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.609762    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-trusted-ca\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.609977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.609804    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-bound-sa-token\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.609977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.609844    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-ca-trust-extracted\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.610223 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.610197    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-ca-trust-extracted\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.610608 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.610586    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-registry-certificates\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.610867 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.610841    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-trusted-ca\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.612094 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.612072    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-image-registry-private-configuration\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.612215 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.612199    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-installation-pull-secrets\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.612327 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.612309    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-registry-tls\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.618453 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.618429    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:15.629740 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.629709    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-bound-sa-token\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.630274 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.630259    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-926zw\" (UniqueName: \"kubernetes.io/projected/d21cf5f3-f982-4ac3-9cda-3074f3309c6e-kube-api-access-926zw\") pod \"image-registry-674fd5846d-bn6zd\" (UID: \"d21cf5f3-f982-4ac3-9cda-3074f3309c6e\") " pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.636145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.636127    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hjc4m"
Oct 30 12:36:15.647839 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.647818    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-f67bl"
Oct 30 12:36:15.723266 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.723239    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:15.741234 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.741209    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:15.929942 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.929917    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hjc4m"]
Oct 30 12:36:15.931503 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:36:15.931475    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0cd6d0a_9b96_449a_846f_c57eb11d02e2.slice/crio-372011124f000ee452bac2f95030b0b79ed35cbb14d674d2a7decd4d9abe8ba4 WatchSource:0}: Error finding container 372011124f000ee452bac2f95030b0b79ed35cbb14d674d2a7decd4d9abe8ba4: Status 404 returned error can't find the container with id 372011124f000ee452bac2f95030b0b79ed35cbb14d674d2a7decd4d9abe8ba4
Oct 30 12:36:15.965756 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:15.965718    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-f67bl"]
Oct 30 12:36:15.968646 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:36:15.968607    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93f8976_4436_4f2f_afd7_adc79bd9e975.slice/crio-bd696106749561ab447cc95347bbf2605b3ff81965fa96547c779c8090876a0e WatchSource:0}: Error finding container bd696106749561ab447cc95347bbf2605b3ff81965fa96547c779c8090876a0e: Status 404 returned error can't find the container with id bd696106749561ab447cc95347bbf2605b3ff81965fa96547c779c8090876a0e
Oct 30 12:36:16.036393 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:16.036343    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-674fd5846d-bn6zd"]
Oct 30 12:36:16.038025 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:36:16.037998    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21cf5f3_f982_4ac3_9cda_3074f3309c6e.slice/crio-6c8459265a0f320049d3b876079bbc59af81681e02e11205c8b1d540eac70e46 WatchSource:0}: Error finding container 6c8459265a0f320049d3b876079bbc59af81681e02e11205c8b1d540eac70e46: Status 404 returned error can't find the container with id 6c8459265a0f320049d3b876079bbc59af81681e02e11205c8b1d540eac70e46
Oct 30 12:36:16.440915 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:16.440875    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-f67bl" event={"ID":"e93f8976-4436-4f2f-afd7-adc79bd9e975","Type":"ContainerStarted","Data":"bd696106749561ab447cc95347bbf2605b3ff81965fa96547c779c8090876a0e"}
Oct 30 12:36:16.442825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:16.442791    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674fd5846d-bn6zd" event={"ID":"d21cf5f3-f982-4ac3-9cda-3074f3309c6e","Type":"ContainerStarted","Data":"ee631e66c6a0db0b7c369816f429934d3a17dc17df71b5c0b0d7e81c87abbfbd"}
Oct 30 12:36:16.442966 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:16.442836    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674fd5846d-bn6zd" event={"ID":"d21cf5f3-f982-4ac3-9cda-3074f3309c6e","Type":"ContainerStarted","Data":"6c8459265a0f320049d3b876079bbc59af81681e02e11205c8b1d540eac70e46"}
Oct 30 12:36:16.443490 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:16.443441    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:16.447083 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:16.447063    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hjc4m" event={"ID":"d0cd6d0a-9b96-449a-846f-c57eb11d02e2","Type":"ContainerStarted","Data":"372011124f000ee452bac2f95030b0b79ed35cbb14d674d2a7decd4d9abe8ba4"}
Oct 30 12:36:16.489114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:16.489073    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-674fd5846d-bn6zd" podStartSLOduration=1.489060559 podStartE2EDuration="1.489060559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:16.488561201 +0000 UTC m=+20.303887691" watchObservedRunningTime="2026-10-30 12:36:16.489060559 +0000 UTC m=+20.304387026"
Oct 30 12:36:17.452514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:17.452478    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hjc4m" event={"ID":"d0cd6d0a-9b96-449a-846f-c57eb11d02e2","Type":"ContainerStarted","Data":"b871ef0d2c494ef2ef9a92e44b7985731f668498426549ae5cb5280a44f24b4c"}
Oct 30 12:36:17.501263 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:17.501206    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hjc4m" podStartSLOduration=1.450619756 podStartE2EDuration="2.501187784s" totalImagesPullingTime="1.050568028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:17.488678479 +0000 UTC m=+21.304004967" watchObservedRunningTime="2026-10-30 12:36:17.501187784 +0000 UTC m=+21.316514273"
Oct 30 12:36:18.456298 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:18.456263    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hjc4m"
Oct 30 12:36:18.462815 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:18.462791    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hjc4m"
Oct 30 12:36:18.526414 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:18.526391    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j8d4p"
Oct 30 12:36:18.886606 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:18.886571    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:18.989162 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:18.989127    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:19.047313 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.047286    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"]
Oct 30 12:36:19.051120 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.051089    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.079923 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.079897    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 12:36:19.079923 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.079897    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 12:36:19.080124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.079971    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 12:36:19.092098 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.092054    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 12:36:19.092098 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.092099    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 12:36:19.092280 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.092131    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9hp9c\""
Oct 30 12:36:19.093010 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.092989    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"]
Oct 30 12:36:19.095761 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.095741    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:19.134888 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.134858    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/de74b154-bc96-4ff3-8e9c-1026b658b174-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-j6txf\" (UID: \"de74b154-bc96-4ff3-8e9c-1026b658b174\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.135008 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.134910    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5xvj\" (UniqueName: \"kubernetes.io/projected/de74b154-bc96-4ff3-8e9c-1026b658b174-kube-api-access-j5xvj\") pod \"prometheus-operator-9cd89b74c-j6txf\" (UID: \"de74b154-bc96-4ff3-8e9c-1026b658b174\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.135073 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.135031    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de74b154-bc96-4ff3-8e9c-1026b658b174-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-j6txf\" (UID: \"de74b154-bc96-4ff3-8e9c-1026b658b174\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.135128 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.135099    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de74b154-bc96-4ff3-8e9c-1026b658b174-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-j6txf\" (UID: \"de74b154-bc96-4ff3-8e9c-1026b658b174\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.236193 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.236136    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/de74b154-bc96-4ff3-8e9c-1026b658b174-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-j6txf\" (UID: \"de74b154-bc96-4ff3-8e9c-1026b658b174\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.236193 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.236176    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5xvj\" (UniqueName: \"kubernetes.io/projected/de74b154-bc96-4ff3-8e9c-1026b658b174-kube-api-access-j5xvj\") pod \"prometheus-operator-9cd89b74c-j6txf\" (UID: \"de74b154-bc96-4ff3-8e9c-1026b658b174\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.236370 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.236224    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de74b154-bc96-4ff3-8e9c-1026b658b174-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-j6txf\" (UID: \"de74b154-bc96-4ff3-8e9c-1026b658b174\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.236370 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.236276    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de74b154-bc96-4ff3-8e9c-1026b658b174-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-j6txf\" (UID: \"de74b154-bc96-4ff3-8e9c-1026b658b174\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.237037 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.237018    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de74b154-bc96-4ff3-8e9c-1026b658b174-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-j6txf\" (UID: \"de74b154-bc96-4ff3-8e9c-1026b658b174\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.238841 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.238816    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de74b154-bc96-4ff3-8e9c-1026b658b174-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-j6txf\" (UID: \"de74b154-bc96-4ff3-8e9c-1026b658b174\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.238948 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.238836    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/de74b154-bc96-4ff3-8e9c-1026b658b174-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-j6txf\" (UID: \"de74b154-bc96-4ff3-8e9c-1026b658b174\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.256156 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.256136    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5xvj\" (UniqueName: \"kubernetes.io/projected/de74b154-bc96-4ff3-8e9c-1026b658b174-kube-api-access-j5xvj\") pod \"prometheus-operator-9cd89b74c-j6txf\" (UID: \"de74b154-bc96-4ff3-8e9c-1026b658b174\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.361422 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.361392    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-85ddd6554f-552pp"]
Oct 30 12:36:19.364804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.364787    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.367245 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.367223    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"
Oct 30 12:36:19.381493 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.381472    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 12:36:19.381605 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.381585    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 12:36:19.381661 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.381600    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 12:36:19.384284 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.384268    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tc44r\""
Oct 30 12:36:19.387787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.387733    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 12:36:19.387897 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.387864    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 12:36:19.388474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.388450    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85ddd6554f-552pp"]
Oct 30 12:36:19.437684 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.437650    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-serving-cert\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.437802 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.437694    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-oauth-serving-cert\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.437802 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.437725    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-console-config\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.437802 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.437785    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6302c600-b4c2-49f2-9959-72178b5b07ba-tmp\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.437917 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.437820    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-service-ca\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.437917 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.437850    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9lfd\" (UniqueName: \"kubernetes.io/projected/6302c600-b4c2-49f2-9959-72178b5b07ba-kube-api-access-l9lfd\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.437917 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.437876    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-oauth-config\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.538499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.538474    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6302c600-b4c2-49f2-9959-72178b5b07ba-tmp\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.538921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.538511    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-service-ca\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.538921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.538546    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9lfd\" (UniqueName: \"kubernetes.io/projected/6302c600-b4c2-49f2-9959-72178b5b07ba-kube-api-access-l9lfd\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.538921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.538574    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-oauth-config\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.538921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.538619    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-serving-cert\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.538921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.538650    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-oauth-serving-cert\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.538921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.538677    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-console-config\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.539212 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.538936    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6302c600-b4c2-49f2-9959-72178b5b07ba-tmp\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.539382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.539346    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-service-ca\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.539464 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.539382    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-console-config\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.539574 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.539516    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-oauth-serving-cert\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.541153 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.541128    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-oauth-config\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.541484 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.541466    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-serving-cert\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.557712 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.557691    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9lfd\" (UniqueName: \"kubernetes.io/projected/6302c600-b4c2-49f2-9959-72178b5b07ba-kube-api-access-l9lfd\") pod \"console-85ddd6554f-552pp\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") " pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.673017 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.672992    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-j6txf"]
Oct 30 12:36:19.678260 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.678128    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:19.979261 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:19.979228    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85ddd6554f-552pp"]
Oct 30 12:36:19.980378 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:36:19.980337    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6302c600_b4c2_49f2_9959_72178b5b07ba.slice/crio-a480d20cced132114853d56041d2ad4df02366562c3629cde22bef317a417830 WatchSource:0}: Error finding container a480d20cced132114853d56041d2ad4df02366562c3629cde22bef317a417830: Status 404 returned error can't find the container with id a480d20cced132114853d56041d2ad4df02366562c3629cde22bef317a417830
Oct 30 12:36:20.470815 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:20.470729    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85ddd6554f-552pp" event={"ID":"6302c600-b4c2-49f2-9959-72178b5b07ba","Type":"ContainerStarted","Data":"a480d20cced132114853d56041d2ad4df02366562c3629cde22bef317a417830"}
Oct 30 12:36:20.473746 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:20.473716    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf" event={"ID":"de74b154-bc96-4ff3-8e9c-1026b658b174","Type":"ContainerStarted","Data":"17e21bd773d6d120f1de779dfb89c98762c724c97a6512d03163868a8f2836fb"}
Oct 30 12:36:21.143888 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:36:21.143852    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:36:21.478660 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:21.478629    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf" event={"ID":"de74b154-bc96-4ff3-8e9c-1026b658b174","Type":"ContainerStarted","Data":"27a2ae860c99d81da4f86196a0e3bd232b147af4642d3b36a8e7f5ec81b10b63"}
Oct 30 12:36:21.478824 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:21.478684    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf" event={"ID":"de74b154-bc96-4ff3-8e9c-1026b658b174","Type":"ContainerStarted","Data":"2d580166d6dc4eae529f0ef6eaa5258a094dcc2f48e8c8e76af5e39443e9bc9a"}
Oct 30 12:36:21.520671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:21.520618    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j6txf" podStartSLOduration=0.905887634 podStartE2EDuration="2.52059999s" totalImagesPullingTime="1.614712356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:21.520275248 +0000 UTC m=+25.335601739" watchObservedRunningTime="2026-10-30 12:36:21.52059999 +0000 UTC m=+25.335926481"
Oct 30 12:36:22.061183 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:22.061146    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:23.324524 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.324431    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:23.490410 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.490338    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85ddd6554f-552pp" event={"ID":"6302c600-b4c2-49f2-9959-72178b5b07ba","Type":"ContainerStarted","Data":"b5d45dfbf05d32a41f79782ea2ad642bdd70b35e74f5a02eb52f2dab0bb915b7"}
Oct 30 12:36:23.536404 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.535529    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-85ddd6554f-552pp" podStartSLOduration=1.477154902 podStartE2EDuration="4.535514212s" totalImagesPullingTime="3.05835931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:23.535261989 +0000 UTC m=+27.350588478" watchObservedRunningTime="2026-10-30 12:36:23.535514212 +0000 UTC m=+27.350840700"
Oct 30 12:36:23.545015 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.544985    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:23.637950 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.637879    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:23.655424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.655395    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fxb8p"]
Oct 30 12:36:23.671896 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.671873    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.735554 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.735475    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 12:36:23.735554 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.735511    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-87th4\""
Oct 30 12:36:23.735554 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.735511    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 12:36:23.735798 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.735587    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 12:36:23.738074 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.738041    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:23.773625 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.773593    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06541c95-2fc8-432d-b39f-32434c96ea43-root\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.773765 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.773657    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-wtmp\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.773765 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.773740    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06541c95-2fc8-432d-b39f-32434c96ea43-metrics-client-ca\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.773845 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.773791    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-textfile\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.773845 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.773830    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6m4l\" (UniqueName: \"kubernetes.io/projected/06541c95-2fc8-432d-b39f-32434c96ea43-kube-api-access-q6m4l\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.773928 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.773898    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-accelerators-collector-config\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.773960 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.773941    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-tls\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.774034 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.774010    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06541c95-2fc8-432d-b39f-32434c96ea43-sys\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.774151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.774039    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.836867 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.836833    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:23.875101 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875058    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-wtmp\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.875101 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875106    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06541c95-2fc8-432d-b39f-32434c96ea43-metrics-client-ca\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.875328 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875143    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-textfile\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.875328 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875162    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6m4l\" (UniqueName: \"kubernetes.io/projected/06541c95-2fc8-432d-b39f-32434c96ea43-kube-api-access-q6m4l\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.875328 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875191    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-accelerators-collector-config\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.875328 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875224    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-tls\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.875328 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875245    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-wtmp\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.875328 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875249    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06541c95-2fc8-432d-b39f-32434c96ea43-sys\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.875328 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875292    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06541c95-2fc8-432d-b39f-32434c96ea43-sys\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.875328 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875307    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.875798 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875338    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06541c95-2fc8-432d-b39f-32434c96ea43-root\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.875798 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875437    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06541c95-2fc8-432d-b39f-32434c96ea43-root\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.875798 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875555    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-textfile\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.875953 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875906    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06541c95-2fc8-432d-b39f-32434c96ea43-metrics-client-ca\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.876000 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.875942    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-accelerators-collector-config\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.878107 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.878079    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-tls\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.878107 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.878091    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06541c95-2fc8-432d-b39f-32434c96ea43-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.898099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.898032    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6m4l\" (UniqueName: \"kubernetes.io/projected/06541c95-2fc8-432d-b39f-32434c96ea43-kube-api-access-q6m4l\") pod \"node-exporter-fxb8p\" (UID: \"06541c95-2fc8-432d-b39f-32434c96ea43\") " pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:23.937840 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.937807    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:23.990450 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:23.990415    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fxb8p"
Oct 30 12:36:24.002640 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:36:24.002605    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06541c95_2fc8_432d_b39f_32434c96ea43.slice/crio-b79f191c51b458a448bc9eea3f8ce5c3ddaf694650bfc8921389f2883a042108 WatchSource:0}: Error finding container b79f191c51b458a448bc9eea3f8ce5c3ddaf694650bfc8921389f2883a042108: Status 404 returned error can't find the container with id b79f191c51b458a448bc9eea3f8ce5c3ddaf694650bfc8921389f2883a042108
Oct 30 12:36:24.037207 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:24.037164    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:24.137363 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:24.137290    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:24.497261 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:24.497224    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fxb8p" event={"ID":"06541c95-2fc8-432d-b39f-32434c96ea43","Type":"ContainerStarted","Data":"b79f191c51b458a448bc9eea3f8ce5c3ddaf694650bfc8921389f2883a042108"}
Oct 30 12:36:24.796180 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:24.796086    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:24.896236 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:24.896173    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:25.045362 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:25.045318    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:25.449409 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:25.449334    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x5ck2"
Oct 30 12:36:25.502784 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:25.502751    4522 generic.go:356] "Generic (PLEG): container finished" podID="06541c95-2fc8-432d-b39f-32434c96ea43" containerID="c7198ff047fe129b8b1c270378b6d28af82ef3f0c456a1902627591b0d0037b8" exitCode=0
Oct 30 12:36:25.503237 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:25.502813    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fxb8p" event={"ID":"06541c95-2fc8-432d-b39f-32434c96ea43","Type":"ContainerDied","Data":"c7198ff047fe129b8b1c270378b6d28af82ef3f0c456a1902627591b0d0037b8"}
Oct 30 12:36:26.508595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:26.508539    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fxb8p" event={"ID":"06541c95-2fc8-432d-b39f-32434c96ea43","Type":"ContainerStarted","Data":"78103dee3e594bb6a7070b05a2d65e9311358cda71055d851cafe534052d069b"}
Oct 30 12:36:26.509054 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:26.508628    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fxb8p" event={"ID":"06541c95-2fc8-432d-b39f-32434c96ea43","Type":"ContainerStarted","Data":"402eac7d91aa4118e8ad2a2294db89ca864791c1d2de45cec6e16acc4ac3fafd"}
Oct 30 12:36:26.572259 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:26.572193    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fxb8p" podStartSLOduration=2.516014067 podStartE2EDuration="3.572172207s" totalImagesPullingTime="978.586792ms" totalInitContainerRuntime="77.571348ms" isStatefulPod=true podCreationTimestamp="2026-10-30 12:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:26.559865503 +0000 UTC m=+30.375192012" watchObservedRunningTime="2026-10-30 12:36:26.572172207 +0000 UTC m=+30.387498698"
Oct 30 12:36:26.855517 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:26.855464    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:26.947460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:26.947425    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:27.048092 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:27.048063    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:27.151180 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:27.151098    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:27.649601 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:27.649567    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:27.759921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:27.759884    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:28.034511 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:28.034471    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:28.142529 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:28.142445    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:28.372784 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:28.372717    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rqz8x"
Oct 30 12:36:28.413056 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:28.413027    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:28.513596 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:28.513562    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:28.613777 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:28.613740    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:29.094967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:29.094894    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:29.184116 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:29.184078    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:29.678804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:29.678772    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:29.678976 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:29.678834    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:29.684675 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:29.684645    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:30.486585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:30.486543    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:30.539942 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:30.539913    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:36:30.934208 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:30.934164    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ff964449f-mm44x"]
Oct 30 12:36:30.968747 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:30.968692    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff964449f-mm44x"]
Oct 30 12:36:30.968920 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:30.968813    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.019551 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.019517    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 12:36:31.034777 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.034745    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65ece8ac-3b83-48b5-8a69-05f53f0d787b-tmp\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.034930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.034808    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-config\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.034930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.034861    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-trusted-ca-bundle\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.034930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.034913    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-service-ca\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.035077 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.035009    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mt4q\" (UniqueName: \"kubernetes.io/projected/65ece8ac-3b83-48b5-8a69-05f53f0d787b-kube-api-access-2mt4q\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.035077 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.035065    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-oauth-serving-cert\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.035143 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.035091    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-oauth-config\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.035143 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.035134    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-serving-cert\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.136076 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.136041    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-oauth-config\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.136234 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.136106    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-serving-cert\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.136234 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.136139    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65ece8ac-3b83-48b5-8a69-05f53f0d787b-tmp\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.136234 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.136170    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-config\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.136234 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.136194    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-trusted-ca-bundle\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.136234 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.136219    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-service-ca\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.136504 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.136244    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mt4q\" (UniqueName: \"kubernetes.io/projected/65ece8ac-3b83-48b5-8a69-05f53f0d787b-kube-api-access-2mt4q\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.136504 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.136277    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-oauth-serving-cert\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.141104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.137131    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65ece8ac-3b83-48b5-8a69-05f53f0d787b-tmp\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.141104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.137547    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-service-ca\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.141104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.137873    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-config\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.141104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.137894    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-trusted-ca-bundle\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.141104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.138050    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-oauth-serving-cert\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.142391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.141751    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-serving-cert\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.144571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.144545    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-oauth-config\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.156317 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.156296    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mt4q\" (UniqueName: \"kubernetes.io/projected/65ece8ac-3b83-48b5-8a69-05f53f0d787b-kube-api-access-2mt4q\") pod \"console-5ff964449f-mm44x\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") " pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.294757 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.294717    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:31.619189 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:31.619117    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff964449f-mm44x"]
Oct 30 12:36:31.625345 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:36:31.625318    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ece8ac_3b83_48b5_8a69_05f53f0d787b.slice/crio-9aef435087e310d05bfa1fd4e8d8681dbf8e3aacb1b2f3170c9afcb803f2998f WatchSource:0}: Error finding container 9aef435087e310d05bfa1fd4e8d8681dbf8e3aacb1b2f3170c9afcb803f2998f: Status 404 returned error can't find the container with id 9aef435087e310d05bfa1fd4e8d8681dbf8e3aacb1b2f3170c9afcb803f2998f
Oct 30 12:36:32.544441 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:32.544402    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-f67bl" event={"ID":"e93f8976-4436-4f2f-afd7-adc79bd9e975","Type":"ContainerStarted","Data":"dfd40bcb92960aeec90b72e578ad7d8afa545fa450999e464ec1820172c8f53c"}
Oct 30 12:36:32.545343 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:32.545320    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-f67bl"
Oct 30 12:36:32.549514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:32.549455    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff964449f-mm44x" event={"ID":"65ece8ac-3b83-48b5-8a69-05f53f0d787b","Type":"ContainerStarted","Data":"098405a790cafe4885ad1b5ab55007cd24a398e9e8fcac38750346a9db1bb16e"}
Oct 30 12:36:32.549514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:32.549493    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff964449f-mm44x" event={"ID":"65ece8ac-3b83-48b5-8a69-05f53f0d787b","Type":"ContainerStarted","Data":"9aef435087e310d05bfa1fd4e8d8681dbf8e3aacb1b2f3170c9afcb803f2998f"}
Oct 30 12:36:32.552492 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:32.552474    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-f67bl"
Oct 30 12:36:32.585204 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:32.585161    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-f67bl" podStartSLOduration=1.254814571 podStartE2EDuration="17.585145613s" totalImagesPullingTime="16.330331042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:32.584159348 +0000 UTC m=+36.399485837" watchObservedRunningTime="2026-10-30 12:36:32.585145613 +0000 UTC m=+36.400472103"
Oct 30 12:36:32.628626 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:32.628586    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5ff964449f-mm44x" podStartSLOduration=2.628573997 podStartE2EDuration="2.628573997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:32.627787348 +0000 UTC m=+36.443113839" watchObservedRunningTime="2026-10-30 12:36:32.628573997 +0000 UTC m=+36.443900485"
Oct 30 12:36:33.701931 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:33.701897    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:33.801151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:33.801099    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:34.952831 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:34.952787    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:35.053506 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:35.053463    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:38.098619 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:38.098574    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:38.463321 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:38.463243    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-674fd5846d-bn6zd"
Oct 30 12:36:38.633129 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:38.633049    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:38.748241 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:38.748166    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:38.870138 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:38.870095    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:40.100402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:40.100368    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:40.182683 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:40.182653    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:41.295254 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:41.295210    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:41.295795 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:41.295304    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:41.300535 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:41.300507    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:41.592224 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:41.592164    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:36:41.660085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:41.660046    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85ddd6554f-552pp"]
Oct 30 12:36:44.135814 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:44.135779    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:44.237267 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:44.237238    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:45.811485 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:45.811454    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:45.949070 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:45.949040    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:46.311467 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:46.311430    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:46.408226 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:46.408190    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:46.546265 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:46.546233    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:49.071108 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:49.071075    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:49.155227 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:49.155194    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:51.133045 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:36:51.133010    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:36:57.041148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:57.041108    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:57.142275 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:57.142237    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:58.029294 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:58.029258    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:36:58.120168 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:36:58.120131    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:01.658115 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:01.658084    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-x8599-trltf"]
Oct 30 12:37:01.659023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:01.658980    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-x8599-trltf" podUID="8749c6a2-8a38-412c-9ea0-ed31e4a1f072" containerName="insights-gathering" containerID="cri-o://81d7cacca8ee7bb99299d1f5af39274c44ecbd5cf6d9e27c80ea0112b8e300fd" gracePeriod=30
Oct 30 12:37:01.846903 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:01.846875    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-whg4s-zmnwk"]
Oct 30 12:37:01.861176 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:01.861153    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:01.868440 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:01.868385    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-whg4s-zmnwk"]
Oct 30 12:37:01.957535 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:01.957512    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:37:01.975222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:01.975198    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-service-ca-bundle\") pod \"periodic-gathering-whg4s-zmnwk\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") " pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:01.975339 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:01.975236    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xqg6\" (UniqueName: \"kubernetes.io/projected/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-kube-api-access-8xqg6\") pod \"periodic-gathering-whg4s-zmnwk\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") " pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:01.975339 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:01.975271    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-trusted-ca-bundle\") pod \"periodic-gathering-whg4s-zmnwk\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") " pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:01.975460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:01.975379    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-archives-path\") pod \"periodic-gathering-whg4s-zmnwk\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") " pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:02.076204 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.076172    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-trusted-ca-bundle\") pod \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") "
Oct 30 12:37:02.076204 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.076205    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-service-ca-bundle\") pod \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") "
Oct 30 12:37:02.076438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.076231    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-archives-path\") pod \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") "
Oct 30 12:37:02.076438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.076288    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-kube-api-access-4pwxq\" (UniqueName: \"kubernetes.io/projected/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-kube-api-access-4pwxq\") pod \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\" (UID: \"8749c6a2-8a38-412c-9ea0-ed31e4a1f072\") "
Oct 30 12:37:02.076438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.076392    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqg6\" (UniqueName: \"kubernetes.io/projected/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-kube-api-access-8xqg6\") pod \"periodic-gathering-whg4s-zmnwk\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") " pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:02.076597 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.076454    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-trusted-ca-bundle\") pod \"periodic-gathering-whg4s-zmnwk\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") " pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:02.076597 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.076485    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-archives-path\") pod \"periodic-gathering-whg4s-zmnwk\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") " pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:02.076597 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.076535    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-service-ca-bundle\") pod \"periodic-gathering-whg4s-zmnwk\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") " pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:02.076597 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.076569    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-service-ca-bundle" pod "8749c6a2-8a38-412c-9ea0-ed31e4a1f072" (UID: "8749c6a2-8a38-412c-9ea0-ed31e4a1f072"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:02.076597 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.076588    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-trusted-ca-bundle" pod "8749c6a2-8a38-412c-9ea0-ed31e4a1f072" (UID: "8749c6a2-8a38-412c-9ea0-ed31e4a1f072"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:02.076821 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.076610    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-archives-path" pod "8749c6a2-8a38-412c-9ea0-ed31e4a1f072" (UID: "8749c6a2-8a38-412c-9ea0-ed31e4a1f072"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:02.076921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.076896    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-archives-path\") pod \"periodic-gathering-whg4s-zmnwk\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") " pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:02.077421 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.077400    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-service-ca-bundle\") pod \"periodic-gathering-whg4s-zmnwk\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") " pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:02.077582 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.077566    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-trusted-ca-bundle\") pod \"periodic-gathering-whg4s-zmnwk\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") " pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:02.078610 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.078589    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-kube-api-access-4pwxq" pod "8749c6a2-8a38-412c-9ea0-ed31e4a1f072" (UID: "8749c6a2-8a38-412c-9ea0-ed31e4a1f072"). InnerVolumeSpecName "kube-api-access-4pwxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:02.095691 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.095669    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqg6\" (UniqueName: \"kubernetes.io/projected/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-kube-api-access-8xqg6\") pod \"periodic-gathering-whg4s-zmnwk\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") " pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:02.177831 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.177782    4522 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-trusted-ca-bundle\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:02.177831 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.177804    4522 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-service-ca-bundle\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:02.177831 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.177818    4522 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-archives-path\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:02.177969 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.177832    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pwxq\" (UniqueName: \"kubernetes.io/projected/8749c6a2-8a38-412c-9ea0-ed31e4a1f072-kube-api-access-4pwxq\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:02.187793 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.187764    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:37:02.510568 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.510546    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-whg4s-zmnwk"]
Oct 30 12:37:02.511855 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:37:02.511815    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8315cfb9_4b52_45c4_ba7d_60d39fa0cc1d.slice/crio-dd3d14efd83154626f08c5931c1195cd3875fe2b0bfbc3a79f8f907853131ae5 WatchSource:0}: Error finding container dd3d14efd83154626f08c5931c1195cd3875fe2b0bfbc3a79f8f907853131ae5: Status 404 returned error can't find the container with id dd3d14efd83154626f08c5931c1195cd3875fe2b0bfbc3a79f8f907853131ae5
Oct 30 12:37:02.666642 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.666614    4522 generic.go:356] "Generic (PLEG): container finished" podID="8749c6a2-8a38-412c-9ea0-ed31e4a1f072" containerID="81d7cacca8ee7bb99299d1f5af39274c44ecbd5cf6d9e27c80ea0112b8e300fd" exitCode=2
Oct 30 12:37:02.667051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.666657    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x8599-trltf"
Oct 30 12:37:02.667051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.666696    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x8599-trltf" event={"ID":"8749c6a2-8a38-412c-9ea0-ed31e4a1f072","Type":"ContainerDied","Data":"81d7cacca8ee7bb99299d1f5af39274c44ecbd5cf6d9e27c80ea0112b8e300fd"}
Oct 30 12:37:02.667051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.666751    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x8599-trltf" event={"ID":"8749c6a2-8a38-412c-9ea0-ed31e4a1f072","Type":"ContainerDied","Data":"0c47e7f7f5c18769d4e0b23be2c17147ba1aee8e6477f845a42903b4704b1b8e"}
Oct 30 12:37:02.667051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.666782    4522 scope.go:122] "RemoveContainer" containerID="81d7cacca8ee7bb99299d1f5af39274c44ecbd5cf6d9e27c80ea0112b8e300fd"
Oct 30 12:37:02.668419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.668394    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-whg4s-zmnwk" event={"ID":"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d","Type":"ContainerStarted","Data":"951cbd27922dd26fbc9cfb973024a503cc9580ae5c0789ec1e1831c3342cc16f"}
Oct 30 12:37:02.668515 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.668441    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-whg4s-zmnwk" event={"ID":"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d","Type":"ContainerStarted","Data":"dd3d14efd83154626f08c5931c1195cd3875fe2b0bfbc3a79f8f907853131ae5"}
Oct 30 12:37:02.680935 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.680917    4522 scope.go:122] "RemoveContainer" containerID="81d7cacca8ee7bb99299d1f5af39274c44ecbd5cf6d9e27c80ea0112b8e300fd"
Oct 30 12:37:02.681250 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:37:02.681231    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d7cacca8ee7bb99299d1f5af39274c44ecbd5cf6d9e27c80ea0112b8e300fd\": container with ID starting with 81d7cacca8ee7bb99299d1f5af39274c44ecbd5cf6d9e27c80ea0112b8e300fd not found: ID does not exist" containerID="81d7cacca8ee7bb99299d1f5af39274c44ecbd5cf6d9e27c80ea0112b8e300fd"
Oct 30 12:37:02.681312 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.681258    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d7cacca8ee7bb99299d1f5af39274c44ecbd5cf6d9e27c80ea0112b8e300fd"} err="failed to get container status \"81d7cacca8ee7bb99299d1f5af39274c44ecbd5cf6d9e27c80ea0112b8e300fd\": rpc error: code = NotFound desc = could not find container \"81d7cacca8ee7bb99299d1f5af39274c44ecbd5cf6d9e27c80ea0112b8e300fd\": container with ID starting with 81d7cacca8ee7bb99299d1f5af39274c44ecbd5cf6d9e27c80ea0112b8e300fd not found: ID does not exist"
Oct 30 12:37:02.719288 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.719263    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-x8599-trltf"]
Oct 30 12:37:02.721474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.721456    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-x8599-trltf"]
Oct 30 12:37:02.770874 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:02.770838    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-whg4s-zmnwk" podStartSLOduration=1.770827945 podStartE2EDuration="1.770827945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:02.770775924 +0000 UTC m=+66.586102410" watchObservedRunningTime="2026-10-30 12:37:02.770827945 +0000 UTC m=+66.586154441"
Oct 30 12:37:03.326386 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:03.326339    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pfqxj_bcd4c4a8-7ece-4ff4-b717-6556ec9c9aed/serve-healthcheck-canary/0.log"
Oct 30 12:37:03.937555 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:03.937519    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:04.039913 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:04.039875    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:04.331729 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:04.331699    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8749c6a2-8a38-412c-9ea0-ed31e4a1f072" path="/var/lib/kubelet/pods/8749c6a2-8a38-412c-9ea0-ed31e4a1f072/volumes"
Oct 30 12:37:06.683240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:06.683184    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-85ddd6554f-552pp" podUID="6302c600-b4c2-49f2-9959-72178b5b07ba" containerName="console" containerID="cri-o://b5d45dfbf05d32a41f79782ea2ad642bdd70b35e74f5a02eb52f2dab0bb915b7" gracePeriod=15
Oct 30 12:37:06.960973 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:06.960919    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:37:07.113694 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.113668    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-serving-cert\") pod \"6302c600-b4c2-49f2-9959-72178b5b07ba\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") "
Oct 30 12:37:07.113822 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.113722    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-service-ca\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-service-ca\") pod \"6302c600-b4c2-49f2-9959-72178b5b07ba\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") "
Oct 30 12:37:07.113908 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.113884    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-console-config\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-console-config\") pod \"6302c600-b4c2-49f2-9959-72178b5b07ba\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") "
Oct 30 12:37:07.113964 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.113927    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6302c600-b4c2-49f2-9959-72178b5b07ba-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6302c600-b4c2-49f2-9959-72178b5b07ba-tmp\") pod \"6302c600-b4c2-49f2-9959-72178b5b07ba\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") "
Oct 30 12:37:07.114020 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.113963    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6302c600-b4c2-49f2-9959-72178b5b07ba-kube-api-access-l9lfd\" (UniqueName: \"kubernetes.io/projected/6302c600-b4c2-49f2-9959-72178b5b07ba-kube-api-access-l9lfd\") pod \"6302c600-b4c2-49f2-9959-72178b5b07ba\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") "
Oct 30 12:37:07.114020 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.113997    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-oauth-serving-cert\") pod \"6302c600-b4c2-49f2-9959-72178b5b07ba\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") "
Oct 30 12:37:07.114122 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.114031    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-oauth-config\") pod \"6302c600-b4c2-49f2-9959-72178b5b07ba\" (UID: \"6302c600-b4c2-49f2-9959-72178b5b07ba\") "
Oct 30 12:37:07.114122 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.114065    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-service-ca" pod "6302c600-b4c2-49f2-9959-72178b5b07ba" (UID: "6302c600-b4c2-49f2-9959-72178b5b07ba"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:07.114229 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.114187    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6302c600-b4c2-49f2-9959-72178b5b07ba-tmp" pod "6302c600-b4c2-49f2-9959-72178b5b07ba" (UID: "6302c600-b4c2-49f2-9959-72178b5b07ba"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:07.114283 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.114236    4522 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:07.114326 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.114274    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-console-config" pod "6302c600-b4c2-49f2-9959-72178b5b07ba" (UID: "6302c600-b4c2-49f2-9959-72178b5b07ba"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:07.114492 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.114464    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-oauth-serving-cert" pod "6302c600-b4c2-49f2-9959-72178b5b07ba" (UID: "6302c600-b4c2-49f2-9959-72178b5b07ba"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:07.116190 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.116162    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6302c600-b4c2-49f2-9959-72178b5b07ba-kube-api-access-l9lfd" pod "6302c600-b4c2-49f2-9959-72178b5b07ba" (UID: "6302c600-b4c2-49f2-9959-72178b5b07ba"). InnerVolumeSpecName "kube-api-access-l9lfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:07.116190 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.116167    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-oauth-config" pod "6302c600-b4c2-49f2-9959-72178b5b07ba" (UID: "6302c600-b4c2-49f2-9959-72178b5b07ba"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:07.116316 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.116207    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-serving-cert" pod "6302c600-b4c2-49f2-9959-72178b5b07ba" (UID: "6302c600-b4c2-49f2-9959-72178b5b07ba"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:07.214828 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.214780    4522 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-console-config\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:07.214828 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.214801    4522 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6302c600-b4c2-49f2-9959-72178b5b07ba-tmp\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:07.214828 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.214812    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9lfd\" (UniqueName: \"kubernetes.io/projected/6302c600-b4c2-49f2-9959-72178b5b07ba-kube-api-access-l9lfd\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:07.214828 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.214821    4522 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6302c600-b4c2-49f2-9959-72178b5b07ba-oauth-serving-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:07.214828 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.214830    4522 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-oauth-config\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:07.215068 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.214839    4522 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6302c600-b4c2-49f2-9959-72178b5b07ba-console-serving-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:07.685554 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.685518    4522 generic.go:356] "Generic (PLEG): container finished" podID="6302c600-b4c2-49f2-9959-72178b5b07ba" containerID="b5d45dfbf05d32a41f79782ea2ad642bdd70b35e74f5a02eb52f2dab0bb915b7" exitCode=0
Oct 30 12:37:07.685964 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.685575    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85ddd6554f-552pp"
Oct 30 12:37:07.685964 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.685591    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85ddd6554f-552pp" event={"ID":"6302c600-b4c2-49f2-9959-72178b5b07ba","Type":"ContainerDied","Data":"b5d45dfbf05d32a41f79782ea2ad642bdd70b35e74f5a02eb52f2dab0bb915b7"}
Oct 30 12:37:07.685964 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.685646    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85ddd6554f-552pp" event={"ID":"6302c600-b4c2-49f2-9959-72178b5b07ba","Type":"ContainerDied","Data":"a480d20cced132114853d56041d2ad4df02366562c3629cde22bef317a417830"}
Oct 30 12:37:07.685964 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.685668    4522 scope.go:122] "RemoveContainer" containerID="b5d45dfbf05d32a41f79782ea2ad642bdd70b35e74f5a02eb52f2dab0bb915b7"
Oct 30 12:37:07.694081 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.694058    4522 scope.go:122] "RemoveContainer" containerID="b5d45dfbf05d32a41f79782ea2ad642bdd70b35e74f5a02eb52f2dab0bb915b7"
Oct 30 12:37:07.694464 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:37:07.694340    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d45dfbf05d32a41f79782ea2ad642bdd70b35e74f5a02eb52f2dab0bb915b7\": container with ID starting with b5d45dfbf05d32a41f79782ea2ad642bdd70b35e74f5a02eb52f2dab0bb915b7 not found: ID does not exist" containerID="b5d45dfbf05d32a41f79782ea2ad642bdd70b35e74f5a02eb52f2dab0bb915b7"
Oct 30 12:37:07.694522 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.694476    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d45dfbf05d32a41f79782ea2ad642bdd70b35e74f5a02eb52f2dab0bb915b7"} err="failed to get container status \"b5d45dfbf05d32a41f79782ea2ad642bdd70b35e74f5a02eb52f2dab0bb915b7\": rpc error: code = NotFound desc = could not find container \"b5d45dfbf05d32a41f79782ea2ad642bdd70b35e74f5a02eb52f2dab0bb915b7\": container with ID starting with b5d45dfbf05d32a41f79782ea2ad642bdd70b35e74f5a02eb52f2dab0bb915b7 not found: ID does not exist"
Oct 30 12:37:07.738055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.738033    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85ddd6554f-552pp"]
Oct 30 12:37:07.749923 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:07.749905    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85ddd6554f-552pp"]
Oct 30 12:37:08.331055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:08.331021    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6302c600-b4c2-49f2-9959-72178b5b07ba" path="/var/lib/kubelet/pods/6302c600-b4c2-49f2-9959-72178b5b07ba/volumes"
Oct 30 12:37:09.558899 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:09.558782    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:09.659119 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:09.659090    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:09.746900 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:09.746871    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:13.052037 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.052003    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:13.083538 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.083504    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58cbb5f494-dcllq"]
Oct 30 12:37:13.083826 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.083809    4522 cpu_manager.go:414] "containerMap: removing container" podUID="8749c6a2-8a38-412c-9ea0-ed31e4a1f072" containerName="insights-gathering"
Oct 30 12:37:13.083909 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.083830    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8749c6a2-8a38-412c-9ea0-ed31e4a1f072" containerName="insights-gathering"
Oct 30 12:37:13.083909 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.083847    4522 cpu_manager.go:414] "containerMap: removing container" podUID="6302c600-b4c2-49f2-9959-72178b5b07ba" containerName="console"
Oct 30 12:37:13.083909 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.083854    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6302c600-b4c2-49f2-9959-72178b5b07ba" containerName="console"
Oct 30 12:37:13.084054 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.083939    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8749c6a2-8a38-412c-9ea0-ed31e4a1f072" containerName="insights-gathering"
Oct 30 12:37:13.084054 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.083952    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6302c600-b4c2-49f2-9959-72178b5b07ba" containerName="console"
Oct 30 12:37:13.104958 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.104936    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.115389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.115369    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58cbb5f494-dcllq"]
Oct 30 12:37:13.171313 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.171280    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:13.253746 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.253726    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-trusted-ca-bundle\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.253840 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.253755    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60b26334-fd26-4028-bd09-32be57ec4c65-tmp\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.253840 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.253777    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wp7l\" (UniqueName: \"kubernetes.io/projected/60b26334-fd26-4028-bd09-32be57ec4c65-kube-api-access-7wp7l\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.253911 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.253889    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-service-ca\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.253948 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.253913    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-oauth-serving-cert\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.253948 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.253934    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-console-config\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.254015 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.253957    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-serving-cert\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.254049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.254029    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-oauth-config\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.271546 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.271524    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:13.354502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.354425    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-oauth-config\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.354502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.354462    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-trusted-ca-bundle\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.354502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.354482    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60b26334-fd26-4028-bd09-32be57ec4c65-tmp\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.354816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.354510    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wp7l\" (UniqueName: \"kubernetes.io/projected/60b26334-fd26-4028-bd09-32be57ec4c65-kube-api-access-7wp7l\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.354816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.354550    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-service-ca\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.354816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.354589    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-oauth-serving-cert\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.354816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.354623    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-console-config\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.354816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.354661    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-serving-cert\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.355099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.354889    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60b26334-fd26-4028-bd09-32be57ec4c65-tmp\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.355445 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.355415    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-console-config\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.355557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.355527    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-service-ca\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.355557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.355544    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-oauth-serving-cert\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.355739 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.355724    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-trusted-ca-bundle\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.357616 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.357594    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-serving-cert\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.357724 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.357594    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-oauth-config\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.374033 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.374009    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wp7l\" (UniqueName: \"kubernetes.io/projected/60b26334-fd26-4028-bd09-32be57ec4c65-kube-api-access-7wp7l\") pod \"console-58cbb5f494-dcllq\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") " pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.429158 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.429130    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:13.487544 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.487514    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:13.591560 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.591527    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:13.761977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:13.761911    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58cbb5f494-dcllq"]
Oct 30 12:37:13.766057 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:37:13.763617    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b26334_fd26_4028_bd09_32be57ec4c65.slice/crio-0185c33288d6b3bd9293ab709aa809a6cc205d20786dd63c501e1793d133963c WatchSource:0}: Error finding container 0185c33288d6b3bd9293ab709aa809a6cc205d20786dd63c501e1793d133963c: Status 404 returned error can't find the container with id 0185c33288d6b3bd9293ab709aa809a6cc205d20786dd63c501e1793d133963c
Oct 30 12:37:14.710577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:14.710540    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cbb5f494-dcllq" event={"ID":"60b26334-fd26-4028-bd09-32be57ec4c65","Type":"ContainerStarted","Data":"7c9027771437ea374a9e116638ddebb328309a78bd887edf5780890bd00d0658"}
Oct 30 12:37:14.710981 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:14.710587    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cbb5f494-dcllq" event={"ID":"60b26334-fd26-4028-bd09-32be57ec4c65","Type":"ContainerStarted","Data":"0185c33288d6b3bd9293ab709aa809a6cc205d20786dd63c501e1793d133963c"}
Oct 30 12:37:14.750363 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:14.750319    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58cbb5f494-dcllq" podStartSLOduration=1.750306068 podStartE2EDuration="1.750306068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:14.749866245 +0000 UTC m=+78.565192733" watchObservedRunningTime="2026-10-30 12:37:14.750306068 +0000 UTC m=+78.565632557"
Oct 30 12:37:19.827889 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:19.827843    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:19.915211 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:19.915174    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:21.132387 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:37:21.132340    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:37:22.812151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:22.812109    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58cbb5f494-dcllq"]
Oct 30 12:37:22.850821 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:22.850793    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7657c947ff-t7nzd"]
Oct 30 12:37:22.854155 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:22.854139    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:22.875115 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:22.875074    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7657c947ff-t7nzd"]
Oct 30 12:37:22.941646 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:22.941610    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:23.021502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.021469    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvwkx\" (UniqueName: \"kubernetes.io/projected/bb204ee7-2696-4856-be85-1efba2443439-kube-api-access-lvwkx\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.021660 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.021512    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb204ee7-2696-4856-be85-1efba2443439-console-oauth-config\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.021660 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.021540    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb204ee7-2696-4856-be85-1efba2443439-oauth-serving-cert\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.021660 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.021645    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb204ee7-2696-4856-be85-1efba2443439-tmp\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.021777 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.021682    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb204ee7-2696-4856-be85-1efba2443439-console-config\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.021777 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.021700    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb204ee7-2696-4856-be85-1efba2443439-trusted-ca-bundle\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.021844 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.021786    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb204ee7-2696-4856-be85-1efba2443439-console-serving-cert\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.021844 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.021813    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb204ee7-2696-4856-be85-1efba2443439-service-ca\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.037244 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.037220    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:23.122828 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.122768    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb204ee7-2696-4856-be85-1efba2443439-console-config\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.122828 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.122798    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb204ee7-2696-4856-be85-1efba2443439-trusted-ca-bundle\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.122828 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.122816    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb204ee7-2696-4856-be85-1efba2443439-console-serving-cert\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.123039 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.122961    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb204ee7-2696-4856-be85-1efba2443439-service-ca\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.123039 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.123026    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvwkx\" (UniqueName: \"kubernetes.io/projected/bb204ee7-2696-4856-be85-1efba2443439-kube-api-access-lvwkx\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.123139 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.123074    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb204ee7-2696-4856-be85-1efba2443439-console-oauth-config\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.123139 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.123132    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb204ee7-2696-4856-be85-1efba2443439-oauth-serving-cert\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.123241 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.123185    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb204ee7-2696-4856-be85-1efba2443439-tmp\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.123547 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.123521    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb204ee7-2696-4856-be85-1efba2443439-console-config\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.123643 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.123563    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb204ee7-2696-4856-be85-1efba2443439-tmp\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.123880 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.123856    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb204ee7-2696-4856-be85-1efba2443439-oauth-serving-cert\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.123969 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.123908    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb204ee7-2696-4856-be85-1efba2443439-trusted-ca-bundle\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.124195 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.124174    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb204ee7-2696-4856-be85-1efba2443439-service-ca\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.125378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.125344    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb204ee7-2696-4856-be85-1efba2443439-console-serving-cert\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.125464 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.125345    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb204ee7-2696-4856-be85-1efba2443439-console-oauth-config\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.141289 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.141265    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvwkx\" (UniqueName: \"kubernetes.io/projected/bb204ee7-2696-4856-be85-1efba2443439-kube-api-access-lvwkx\") pod \"console-7657c947ff-t7nzd\" (UID: \"bb204ee7-2696-4856-be85-1efba2443439\") " pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.166281 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.166259    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:23.238257 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.238230    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:23.430055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.429993    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:23.459111 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.459089    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7657c947ff-t7nzd"]
Oct 30 12:37:23.461446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:37:23.461416    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb204ee7_2696_4856_be85_1efba2443439.slice/crio-9256a6eb00ab0f15d8d2030afef20d5c0dbb3555ae68d2089a0036bdfb8ea477 WatchSource:0}: Error finding container 9256a6eb00ab0f15d8d2030afef20d5c0dbb3555ae68d2089a0036bdfb8ea477: Status 404 returned error can't find the container with id 9256a6eb00ab0f15d8d2030afef20d5c0dbb3555ae68d2089a0036bdfb8ea477
Oct 30 12:37:23.744202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.744122    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7657c947ff-t7nzd" event={"ID":"bb204ee7-2696-4856-be85-1efba2443439","Type":"ContainerStarted","Data":"9243979dd53e394bd3f85f81a4032fe5b86b2d2cc4db040012c01e2e7e1acfb5"}
Oct 30 12:37:23.744202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.744171    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7657c947ff-t7nzd" event={"ID":"bb204ee7-2696-4856-be85-1efba2443439","Type":"ContainerStarted","Data":"9256a6eb00ab0f15d8d2030afef20d5c0dbb3555ae68d2089a0036bdfb8ea477"}
Oct 30 12:37:23.787465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:23.787410    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7657c947ff-t7nzd" podStartSLOduration=1.787391254 podStartE2EDuration="1.787391254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:23.786298311 +0000 UTC m=+87.601624824" watchObservedRunningTime="2026-10-30 12:37:23.787391254 +0000 UTC m=+87.602717742"
Oct 30 12:37:33.167080 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:33.167037    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:33.167720 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:33.167121    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:33.171699 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:33.171675    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:33.780873 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:33.780842    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7657c947ff-t7nzd"
Oct 30 12:37:33.852241 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:33.852211    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff964449f-mm44x"]
Oct 30 12:37:34.363063 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:34.363031    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:34.462982 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:34.462954    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:43.034445 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:43.034401    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:43.134852 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:43.134816    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:45.915760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:45.915720    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:47.830930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:47.830895    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-58cbb5f494-dcllq" podUID="60b26334-fd26-4028-bd09-32be57ec4c65" containerName="console" containerID="cri-o://7c9027771437ea374a9e116638ddebb328309a78bd887edf5780890bd00d0658" gracePeriod=15
Oct 30 12:37:48.066920 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.066899    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:48.100621 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.100577    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-oauth-config\") pod \"60b26334-fd26-4028-bd09-32be57ec4c65\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") "
Oct 30 12:37:48.100621 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.100611    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-oauth-serving-cert\") pod \"60b26334-fd26-4028-bd09-32be57ec4c65\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") "
Oct 30 12:37:48.100744 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.100633    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-service-ca\") pod \"60b26334-fd26-4028-bd09-32be57ec4c65\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") "
Oct 30 12:37:48.100744 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.100659    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-serving-cert\") pod \"60b26334-fd26-4028-bd09-32be57ec4c65\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") "
Oct 30 12:37:48.100744 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.100699    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b26334-fd26-4028-bd09-32be57ec4c65-tmp\" (UniqueName: \"kubernetes.io/empty-dir/60b26334-fd26-4028-bd09-32be57ec4c65-tmp\") pod \"60b26334-fd26-4028-bd09-32be57ec4c65\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") "
Oct 30 12:37:48.100744 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.100722    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-console-config\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-console-config\") pod \"60b26334-fd26-4028-bd09-32be57ec4c65\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") "
Oct 30 12:37:48.100944 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.100748    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b26334-fd26-4028-bd09-32be57ec4c65-kube-api-access-7wp7l\" (UniqueName: \"kubernetes.io/projected/60b26334-fd26-4028-bd09-32be57ec4c65-kube-api-access-7wp7l\") pod \"60b26334-fd26-4028-bd09-32be57ec4c65\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") "
Oct 30 12:37:48.100944 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.100802    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-trusted-ca-bundle\") pod \"60b26334-fd26-4028-bd09-32be57ec4c65\" (UID: \"60b26334-fd26-4028-bd09-32be57ec4c65\") "
Oct 30 12:37:48.101044 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.101011    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b26334-fd26-4028-bd09-32be57ec4c65-tmp" pod "60b26334-fd26-4028-bd09-32be57ec4c65" (UID: "60b26334-fd26-4028-bd09-32be57ec4c65"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:48.101044 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.101032    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-service-ca" pod "60b26334-fd26-4028-bd09-32be57ec4c65" (UID: "60b26334-fd26-4028-bd09-32be57ec4c65"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:48.101347 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.101125    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-console-config" pod "60b26334-fd26-4028-bd09-32be57ec4c65" (UID: "60b26334-fd26-4028-bd09-32be57ec4c65"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:48.101347 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.101296    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-oauth-serving-cert" pod "60b26334-fd26-4028-bd09-32be57ec4c65" (UID: "60b26334-fd26-4028-bd09-32be57ec4c65"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:48.101347 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.101317    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-trusted-ca-bundle" pod "60b26334-fd26-4028-bd09-32be57ec4c65" (UID: "60b26334-fd26-4028-bd09-32be57ec4c65"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:48.102867 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.102840    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b26334-fd26-4028-bd09-32be57ec4c65-kube-api-access-7wp7l" pod "60b26334-fd26-4028-bd09-32be57ec4c65" (UID: "60b26334-fd26-4028-bd09-32be57ec4c65"). InnerVolumeSpecName "kube-api-access-7wp7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:48.103122 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.103096    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-serving-cert" pod "60b26334-fd26-4028-bd09-32be57ec4c65" (UID: "60b26334-fd26-4028-bd09-32be57ec4c65"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:48.103186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.103114    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-oauth-config" pod "60b26334-fd26-4028-bd09-32be57ec4c65" (UID: "60b26334-fd26-4028-bd09-32be57ec4c65"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:48.202019 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.201997    4522 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-serving-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:48.202019 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.202018    4522 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60b26334-fd26-4028-bd09-32be57ec4c65-tmp\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:48.202148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.202027    4522 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-console-config\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:48.202148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.202037    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wp7l\" (UniqueName: \"kubernetes.io/projected/60b26334-fd26-4028-bd09-32be57ec4c65-kube-api-access-7wp7l\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:48.202148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.202045    4522 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-trusted-ca-bundle\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:48.202148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.202053    4522 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60b26334-fd26-4028-bd09-32be57ec4c65-console-oauth-config\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:48.202148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.202062    4522 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-oauth-serving-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:48.202148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.202070    4522 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60b26334-fd26-4028-bd09-32be57ec4c65-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:48.828179 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.828146    4522 generic.go:356] "Generic (PLEG): container finished" podID="60b26334-fd26-4028-bd09-32be57ec4c65" containerID="7c9027771437ea374a9e116638ddebb328309a78bd887edf5780890bd00d0658" exitCode=0
Oct 30 12:37:48.828304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.828204    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cbb5f494-dcllq"
Oct 30 12:37:48.828304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.828213    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cbb5f494-dcllq" event={"ID":"60b26334-fd26-4028-bd09-32be57ec4c65","Type":"ContainerDied","Data":"7c9027771437ea374a9e116638ddebb328309a78bd887edf5780890bd00d0658"}
Oct 30 12:37:48.828304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.828279    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cbb5f494-dcllq" event={"ID":"60b26334-fd26-4028-bd09-32be57ec4c65","Type":"ContainerDied","Data":"0185c33288d6b3bd9293ab709aa809a6cc205d20786dd63c501e1793d133963c"}
Oct 30 12:37:48.828304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.828299    4522 scope.go:122] "RemoveContainer" containerID="7c9027771437ea374a9e116638ddebb328309a78bd887edf5780890bd00d0658"
Oct 30 12:37:48.836171 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.836025    4522 scope.go:122] "RemoveContainer" containerID="7c9027771437ea374a9e116638ddebb328309a78bd887edf5780890bd00d0658"
Oct 30 12:37:48.836483 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:37:48.836312    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9027771437ea374a9e116638ddebb328309a78bd887edf5780890bd00d0658\": container with ID starting with 7c9027771437ea374a9e116638ddebb328309a78bd887edf5780890bd00d0658 not found: ID does not exist" containerID="7c9027771437ea374a9e116638ddebb328309a78bd887edf5780890bd00d0658"
Oct 30 12:37:48.836483 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.836337    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9027771437ea374a9e116638ddebb328309a78bd887edf5780890bd00d0658"} err="failed to get container status \"7c9027771437ea374a9e116638ddebb328309a78bd887edf5780890bd00d0658\": rpc error: code = NotFound desc = could not find container \"7c9027771437ea374a9e116638ddebb328309a78bd887edf5780890bd00d0658\": container with ID starting with 7c9027771437ea374a9e116638ddebb328309a78bd887edf5780890bd00d0658 not found: ID does not exist"
Oct 30 12:37:48.876760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.876735    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58cbb5f494-dcllq"]
Oct 30 12:37:48.888417 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:48.888397    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58cbb5f494-dcllq"]
Oct 30 12:37:50.331943 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:50.331908    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b26334-fd26-4028-bd09-32be57ec4c65" path="/var/lib/kubelet/pods/60b26334-fd26-4028-bd09-32be57ec4c65/volumes"
Oct 30 12:37:51.132897 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:37:51.132865    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:37:52.118838 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:52.118806    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:52.218231 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:52.218200    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:37:58.873406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:58.873345    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5ff964449f-mm44x" podUID="65ece8ac-3b83-48b5-8a69-05f53f0d787b" containerName="console" containerID="cri-o://098405a790cafe4885ad1b5ab55007cd24a398e9e8fcac38750346a9db1bb16e" gracePeriod=15
Oct 30 12:37:59.115230 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.115204    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:37:59.179738 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.179661    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ece8ac-3b83-48b5-8a69-05f53f0d787b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/65ece8ac-3b83-48b5-8a69-05f53f0d787b-tmp\") pod \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") "
Oct 30 12:37:59.179738 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.179711    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-serving-cert\") pod \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") "
Oct 30 12:37:59.179903 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.179853    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ece8ac-3b83-48b5-8a69-05f53f0d787b-kube-api-access-2mt4q\" (UniqueName: \"kubernetes.io/projected/65ece8ac-3b83-48b5-8a69-05f53f0d787b-kube-api-access-2mt4q\") pod \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") "
Oct 30 12:37:59.179903 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.179892    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-config\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-config\") pod \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") "
Oct 30 12:37:59.179997 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.179912    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-oauth-serving-cert\") pod \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") "
Oct 30 12:37:59.179997 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.179936    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ece8ac-3b83-48b5-8a69-05f53f0d787b-tmp" pod "65ece8ac-3b83-48b5-8a69-05f53f0d787b" (UID: "65ece8ac-3b83-48b5-8a69-05f53f0d787b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:59.180100 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.180018    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-service-ca\") pod \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") "
Oct 30 12:37:59.180100 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.180077    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-oauth-config\") pod \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") "
Oct 30 12:37:59.180197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.180132    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-trusted-ca-bundle\") pod \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\" (UID: \"65ece8ac-3b83-48b5-8a69-05f53f0d787b\") "
Oct 30 12:37:59.180266 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.180244    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-oauth-serving-cert" pod "65ece8ac-3b83-48b5-8a69-05f53f0d787b" (UID: "65ece8ac-3b83-48b5-8a69-05f53f0d787b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:59.180266 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.180253    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-config" pod "65ece8ac-3b83-48b5-8a69-05f53f0d787b" (UID: "65ece8ac-3b83-48b5-8a69-05f53f0d787b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:59.180377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.180319    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-service-ca" pod "65ece8ac-3b83-48b5-8a69-05f53f0d787b" (UID: "65ece8ac-3b83-48b5-8a69-05f53f0d787b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:59.180543 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.180519    4522 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65ece8ac-3b83-48b5-8a69-05f53f0d787b-tmp\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:59.180665 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.180546    4522 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-config\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:59.180665 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.180561    4522 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-oauth-serving-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:59.180665 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.180575    4522 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:59.180665 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.180612    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-trusted-ca-bundle" pod "65ece8ac-3b83-48b5-8a69-05f53f0d787b" (UID: "65ece8ac-3b83-48b5-8a69-05f53f0d787b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:59.181802 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.181773    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-serving-cert" pod "65ece8ac-3b83-48b5-8a69-05f53f0d787b" (UID: "65ece8ac-3b83-48b5-8a69-05f53f0d787b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:59.181889 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.181876    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ece8ac-3b83-48b5-8a69-05f53f0d787b-kube-api-access-2mt4q" pod "65ece8ac-3b83-48b5-8a69-05f53f0d787b" (UID: "65ece8ac-3b83-48b5-8a69-05f53f0d787b"). InnerVolumeSpecName "kube-api-access-2mt4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:59.182095 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.182073    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-oauth-config" pod "65ece8ac-3b83-48b5-8a69-05f53f0d787b" (UID: "65ece8ac-3b83-48b5-8a69-05f53f0d787b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:37:59.281822 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.281790    4522 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-serving-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:59.281822 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.281815    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mt4q\" (UniqueName: \"kubernetes.io/projected/65ece8ac-3b83-48b5-8a69-05f53f0d787b-kube-api-access-2mt4q\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:59.281822 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.281826    4522 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65ece8ac-3b83-48b5-8a69-05f53f0d787b-console-oauth-config\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:59.282019 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.281835    4522 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ece8ac-3b83-48b5-8a69-05f53f0d787b-trusted-ca-bundle\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:37:59.866563 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.866523    4522 generic.go:356] "Generic (PLEG): container finished" podID="65ece8ac-3b83-48b5-8a69-05f53f0d787b" containerID="098405a790cafe4885ad1b5ab55007cd24a398e9e8fcac38750346a9db1bb16e" exitCode=0
Oct 30 12:37:59.866736 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.866571    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff964449f-mm44x"
Oct 30 12:37:59.866736 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.866604    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff964449f-mm44x" event={"ID":"65ece8ac-3b83-48b5-8a69-05f53f0d787b","Type":"ContainerDied","Data":"098405a790cafe4885ad1b5ab55007cd24a398e9e8fcac38750346a9db1bb16e"}
Oct 30 12:37:59.866736 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.866653    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff964449f-mm44x" event={"ID":"65ece8ac-3b83-48b5-8a69-05f53f0d787b","Type":"ContainerDied","Data":"9aef435087e310d05bfa1fd4e8d8681dbf8e3aacb1b2f3170c9afcb803f2998f"}
Oct 30 12:37:59.866736 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.866677    4522 scope.go:122] "RemoveContainer" containerID="098405a790cafe4885ad1b5ab55007cd24a398e9e8fcac38750346a9db1bb16e"
Oct 30 12:37:59.875124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.874913    4522 scope.go:122] "RemoveContainer" containerID="098405a790cafe4885ad1b5ab55007cd24a398e9e8fcac38750346a9db1bb16e"
Oct 30 12:37:59.875381 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:37:59.875187    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"098405a790cafe4885ad1b5ab55007cd24a398e9e8fcac38750346a9db1bb16e\": container with ID starting with 098405a790cafe4885ad1b5ab55007cd24a398e9e8fcac38750346a9db1bb16e not found: ID does not exist" containerID="098405a790cafe4885ad1b5ab55007cd24a398e9e8fcac38750346a9db1bb16e"
Oct 30 12:37:59.875381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.875212    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"098405a790cafe4885ad1b5ab55007cd24a398e9e8fcac38750346a9db1bb16e"} err="failed to get container status \"098405a790cafe4885ad1b5ab55007cd24a398e9e8fcac38750346a9db1bb16e\": rpc error: code = NotFound desc = could not find container \"098405a790cafe4885ad1b5ab55007cd24a398e9e8fcac38750346a9db1bb16e\": container with ID starting with 098405a790cafe4885ad1b5ab55007cd24a398e9e8fcac38750346a9db1bb16e not found: ID does not exist"
Oct 30 12:37:59.918690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.918662    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff964449f-mm44x"]
Oct 30 12:37:59.930138 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:37:59.930109    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ff964449f-mm44x"]
Oct 30 12:38:00.330801 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:00.330768    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ece8ac-3b83-48b5-8a69-05f53f0d787b" path="/var/lib/kubelet/pods/65ece8ac-3b83-48b5-8a69-05f53f0d787b/volumes"
Oct 30 12:38:21.132345 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:38:21.132301    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:38:34.989456 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:34.989371    4522 generic.go:356] "Generic (PLEG): container finished" podID="8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d" containerID="951cbd27922dd26fbc9cfb973024a503cc9580ae5c0789ec1e1831c3342cc16f" exitCode=0
Oct 30 12:38:34.989456 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:34.989446    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-whg4s-zmnwk" event={"ID":"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d","Type":"ContainerDied","Data":"951cbd27922dd26fbc9cfb973024a503cc9580ae5c0789ec1e1831c3342cc16f"}
Oct 30 12:38:36.115654 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.115628    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:38:36.253191 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.253111    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-kube-api-access-8xqg6\" (UniqueName: \"kubernetes.io/projected/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-kube-api-access-8xqg6\") pod \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") "
Oct 30 12:38:36.253191 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.253169    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-trusted-ca-bundle\") pod \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") "
Oct 30 12:38:36.253409 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.253202    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-archives-path\") pod \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") "
Oct 30 12:38:36.253409 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.253226    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-service-ca-bundle\") pod \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\" (UID: \"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d\") "
Oct 30 12:38:36.253564 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.253542    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-archives-path" pod "8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d" (UID: "8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:38:36.253633 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.253582    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-trusted-ca-bundle" pod "8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d" (UID: "8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:38:36.253734 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.253708    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-service-ca-bundle" pod "8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d" (UID: "8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:38:36.255192 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.255169    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-kube-api-access-8xqg6" pod "8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d" (UID: "8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d"). InnerVolumeSpecName "kube-api-access-8xqg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:38:36.353786 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.353757    4522 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-archives-path\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:38:36.353786 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.353784    4522 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-service-ca-bundle\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:38:36.353948 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.353795    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xqg6\" (UniqueName: \"kubernetes.io/projected/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-kube-api-access-8xqg6\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:38:36.353948 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.353803    4522 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d-trusted-ca-bundle\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:38:36.996790 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.996763    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-whg4s-zmnwk"
Oct 30 12:38:36.996958 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.996758    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-whg4s-zmnwk" event={"ID":"8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d","Type":"ContainerDied","Data":"dd3d14efd83154626f08c5931c1195cd3875fe2b0bfbc3a79f8f907853131ae5"}
Oct 30 12:38:36.996958 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:36.996903    4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd3d14efd83154626f08c5931c1195cd3875fe2b0bfbc3a79f8f907853131ae5"
Oct 30 12:38:37.395629 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:37.395600    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:37.472137 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:37.472108    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:46.038386 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:38:46.038334    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:38:51.135598 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:38:51.135553    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:39:01.870374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:01.870313    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:01.944601 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:01.944572    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:04.115396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:04.115339    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:21.133801 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:39:21.133760    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:39:22.424948 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:22.424916    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:22.548622 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:22.548596    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:22.660186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:22.660154    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:23.767594 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.767564    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"]
Oct 30 12:39:23.767954 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.767802    4522 cpu_manager.go:414] "containerMap: removing container" podUID="65ece8ac-3b83-48b5-8a69-05f53f0d787b" containerName="console"
Oct 30 12:39:23.767954 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.767812    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ece8ac-3b83-48b5-8a69-05f53f0d787b" containerName="console"
Oct 30 12:39:23.767954 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.767828    4522 cpu_manager.go:414] "containerMap: removing container" podUID="8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d" containerName="insights-gathering"
Oct 30 12:39:23.767954 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.767832    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d" containerName="insights-gathering"
Oct 30 12:39:23.767954 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.767843    4522 cpu_manager.go:414] "containerMap: removing container" podUID="60b26334-fd26-4028-bd09-32be57ec4c65" containerName="console"
Oct 30 12:39:23.767954 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.767849    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b26334-fd26-4028-bd09-32be57ec4c65" containerName="console"
Oct 30 12:39:23.767954 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.767891    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8315cfb9-4b52-45c4-ba7d-60d39fa0cc1d" containerName="insights-gathering"
Oct 30 12:39:23.767954 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.767899    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b26334-fd26-4028-bd09-32be57ec4c65" containerName="console"
Oct 30 12:39:23.767954 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.767907    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ece8ac-3b83-48b5-8a69-05f53f0d787b" containerName="console"
Oct 30 12:39:23.770736 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.770713    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.775379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.775338    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:23.787196 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.787171    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 12:39:23.787287 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.787178    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 12:39:23.787287 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.787182    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 12:39:23.787287 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.787185    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 12:39:23.787287 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.787229    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jvqff\""
Oct 30 12:39:23.787287 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.787230    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 12:39:23.798946 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.798924    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 12:39:23.800116 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.800094    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"]
Oct 30 12:39:23.872099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.872074    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.872192 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.872113    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.872192 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.872149    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5pvq\" (UniqueName: \"kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-kube-api-access-c5pvq\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.872271 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.872245    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.872319 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.872299    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.872383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.872330    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb92ec6-6839-4333-a661-e7d8d305cf29-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.872434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.872424    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.878841 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.878819    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:23.973400 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.973374    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5pvq\" (UniqueName: \"kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-kube-api-access-c5pvq\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.973496 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.973407    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.973557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.973519    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.973613 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.973556    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb92ec6-6839-4333-a661-e7d8d305cf29-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.973673 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.973612    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.973779 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.973755    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.973853 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.973810    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.974238 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.974216    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.975701 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.975682    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb92ec6-6839-4333-a661-e7d8d305cf29-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.975934 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.975915    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.976099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.976083    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.976213 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.976194    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.991968 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.991945    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:23.992213 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:23.992196    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5pvq\" (UniqueName: \"kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-kube-api-access-c5pvq\") pod \"istiod-openshift-gateway-7876bfcb74-b7nrt\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:24.081563 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:24.081543    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:24.385734 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:24.385709    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"]
Oct 30 12:39:24.386568 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:39:24.386542    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb92ec6_6839_4333_a661_e7d8d305cf29.slice/crio-1e48f8199c410453024fc256884ab8618c84b8661a799719e621578d1bfccba6 WatchSource:0}: Error finding container 1e48f8199c410453024fc256884ab8618c84b8661a799719e621578d1bfccba6: Status 404 returned error can't find the container with id 1e48f8199c410453024fc256884ab8618c84b8661a799719e621578d1bfccba6
Oct 30 12:39:24.812945 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:24.812839    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:25.152512 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:25.152428    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt" event={"ID":"cfb92ec6-6839-4333-a661-e7d8d305cf29","Type":"ContainerStarted","Data":"1e48f8199c410453024fc256884ab8618c84b8661a799719e621578d1bfccba6"}
Oct 30 12:39:26.508445 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:26.508416    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:39:26.508692 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:26.508482    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:39:27.161544 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:27.161501    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt" event={"ID":"cfb92ec6-6839-4333-a661-e7d8d305cf29","Type":"ContainerStarted","Data":"2be29610aba89410501a78d3d6adadd485b5054ca25787255969f8ea2cb7b4fa"}
Oct 30 12:39:27.161752 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:27.161731    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:27.163224 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:27.163194    4522 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-b7nrt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:39:27.163365 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:27.163247    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt" podUID="cfb92ec6-6839-4333-a661-e7d8d305cf29" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:39:27.219133 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:27.219100    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:27.219390 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:27.219324    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt" podStartSLOduration=2.100079739 podStartE2EDuration="4.219307493s" totalImagesPullingTime="2.119227754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:27.218937757 +0000 UTC m=+211.034264250" watchObservedRunningTime="2026-10-30 12:39:27.219307493 +0000 UTC m=+211.034633983"
Oct 30 12:39:27.302515 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:27.302477    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:27.962132 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:27.962102    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:28.167998 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:28.167967    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:39:29.139212 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:29.139169    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:31.642419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:31.642376    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:46.176415 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:46.176327    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:51.136002 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:39:51.135963    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:39:52.770464 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:52.770429    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:53.716333 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:53.716285    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:53.816289 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:53.816261    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:53.946536 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:53.946515    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:54.331171 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:54.331137    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:57.263943 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:57.263912    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:57.373093 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:57.373059    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:57.789181 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:57.789146    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:58.058961 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:58.058896    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:59.148553 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.148518    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:59.350865 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.350826    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"]
Oct 30 12:39:59.361279 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.361252    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.378130 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.378100    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"]
Oct 30 12:39:59.415171 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.415114    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/febd0b85-6ce3-4244-85ea-6c67d3208953-metrics-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.415171 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.415150    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs5pt\" (UniqueName: \"kubernetes.io/projected/febd0b85-6ce3-4244-85ea-6c67d3208953-kube-api-access-xs5pt\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.415301 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.415234    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/febd0b85-6ce3-4244-85ea-6c67d3208953-manifests\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.415348 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.415313    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/febd0b85-6ce3-4244-85ea-6c67d3208953-apiservice-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.415348 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.415335    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/febd0b85-6ce3-4244-85ea-6c67d3208953-webhook-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.415540 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.415403    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/febd0b85-6ce3-4244-85ea-6c67d3208953-tmp\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.428749 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.428730    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 12:39:59.428890 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.428876    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 12:39:59.429013 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.428953    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 12:39:59.429209 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.429187    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-r2gbl\""
Oct 30 12:39:59.446913 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.446896    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 12:39:59.447081 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.447068    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 12:39:59.506855 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.506832    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:59.516873 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.516843    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/febd0b85-6ce3-4244-85ea-6c67d3208953-tmp\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.516967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.516890    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/febd0b85-6ce3-4244-85ea-6c67d3208953-metrics-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.516967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.516910    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5pt\" (UniqueName: \"kubernetes.io/projected/febd0b85-6ce3-4244-85ea-6c67d3208953-kube-api-access-xs5pt\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.516967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.516938    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/febd0b85-6ce3-4244-85ea-6c67d3208953-manifests\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.517084 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.516976    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/febd0b85-6ce3-4244-85ea-6c67d3208953-apiservice-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.517084 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.517001    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/febd0b85-6ce3-4244-85ea-6c67d3208953-webhook-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.517335 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.517316    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/febd0b85-6ce3-4244-85ea-6c67d3208953-manifests\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.519333 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.519306    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/febd0b85-6ce3-4244-85ea-6c67d3208953-tmp\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.519471 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.519450    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/febd0b85-6ce3-4244-85ea-6c67d3208953-webhook-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.519533 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.519456    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/febd0b85-6ce3-4244-85ea-6c67d3208953-apiservice-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.519533 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.519515    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/febd0b85-6ce3-4244-85ea-6c67d3208953-metrics-cert\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.551758 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.551738    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5pt\" (UniqueName: \"kubernetes.io/projected/febd0b85-6ce3-4244-85ea-6c67d3208953-kube-api-access-xs5pt\") pod \"opendatahub-operator-controller-manager-786cc47469-kh25x\" (UID: \"febd0b85-6ce3-4244-85ea-6c67d3208953\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.683047 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.683000    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:39:59.701138 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.701115    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:59.839313 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.839282    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:39:59.981323 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:39:59.981261    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:00.055673 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:00.055650    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"]
Oct 30 12:40:00.055797 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:40:00.055767    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebd0b85_6ce3_4244_85ea_6c67d3208953.slice/crio-9b44a527fb7abfc74bf9700dae72290d9612bade413fc07c0b36d0ccee71e9bc WatchSource:0}: Error finding container 9b44a527fb7abfc74bf9700dae72290d9612bade413fc07c0b36d0ccee71e9bc: Status 404 returned error can't find the container with id 9b44a527fb7abfc74bf9700dae72290d9612bade413fc07c0b36d0ccee71e9bc
Oct 30 12:40:00.163151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:00.163120    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:00.271987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:00.271958    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x" event={"ID":"febd0b85-6ce3-4244-85ea-6c67d3208953","Type":"ContainerStarted","Data":"9b44a527fb7abfc74bf9700dae72290d9612bade413fc07c0b36d0ccee71e9bc"}
Oct 30 12:40:02.129785 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:02.129746    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:02.312573 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:02.312533    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:02.445387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:02.445292    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:02.879553 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:02.879529    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:03.284270 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:03.284243    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x" event={"ID":"febd0b85-6ce3-4244-85ea-6c67d3208953","Type":"ContainerStarted","Data":"df95c89ddbd643dd1bd8fe043ede496a7da40315198607a901fca0d8b3b213d6"}
Oct 30 12:40:04.290840 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:04.290800    4522 generic.go:356] "Generic (PLEG): container finished" podID="febd0b85-6ce3-4244-85ea-6c67d3208953" containerID="df95c89ddbd643dd1bd8fe043ede496a7da40315198607a901fca0d8b3b213d6" exitCode=0
Oct 30 12:40:04.291290 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:04.290887    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x" event={"ID":"febd0b85-6ce3-4244-85ea-6c67d3208953","Type":"ContainerDied","Data":"df95c89ddbd643dd1bd8fe043ede496a7da40315198607a901fca0d8b3b213d6"}
Oct 30 12:40:04.291290 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:04.291014    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:40:04.291290 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:04.291041    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x" event={"ID":"febd0b85-6ce3-4244-85ea-6c67d3208953","Type":"ContainerStarted","Data":"f9ea38851b0bcef5cffcb991bc410563aaa48f9160fe9368a2986a48d263866d"}
Oct 30 12:40:04.389948 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:04.389892    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x" podStartSLOduration=1.885520254 podStartE2EDuration="5.389875732s" totalImagesPullingTime="3.049985764s" totalInitContainerRuntime="454.369714ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:04.377462727 +0000 UTC m=+248.192789213" watchObservedRunningTime="2026-10-30 12:40:04.389875732 +0000 UTC m=+248.205202222"
Oct 30 12:40:05.807653 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:05.807614    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:06.041302 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:06.041269    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:06.187240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:06.187167    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:06.698589 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:06.698559    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:06.815013 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:06.814972    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:07.696068 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:07.696036    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:08.610060 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:08.610030    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:09.757111 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:09.757075    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:10.070523 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:10.070497    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:12.295901 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:12.295866    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:15.296647 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:15.296617    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-786cc47469-kh25x"
Oct 30 12:40:21.132872 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:40:21.132840    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:40:23.642115 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:23.642080    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:24.473512 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:24.473459    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:24.751526 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:24.751453    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:25.697003 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:25.696969    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:25.813155 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:25.813124    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:25.920333 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:25.920304    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:46.281997 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:46.281941    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:51.132479 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:40:51.132445    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:40:53.723396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:53.723338    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:53.874855 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:53.874821    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:54.117779 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.117749    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:54.407787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.407702    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"]
Oct 30 12:40:54.411174 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.411153    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"
Oct 30 12:40:54.416194 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.416170    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:54.431026 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.431001    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 12:40:54.431026 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.431006    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 12:40:54.432517 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.432499    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jkbzr\""
Oct 30 12:40:54.444787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.444763    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"]
Oct 30 12:40:54.580742 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.580713    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78mqb\" (UniqueName: \"kubernetes.io/projected/01215889-00f7-4cfd-97cd-dee00355f9b9-kube-api-access-78mqb\") pod \"kube-auth-proxy-dc45784b6-l4s7t\" (UID: \"01215889-00f7-4cfd-97cd-dee00355f9b9\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"
Oct 30 12:40:54.580879 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.580755    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01215889-00f7-4cfd-97cd-dee00355f9b9-tls-certs\") pod \"kube-auth-proxy-dc45784b6-l4s7t\" (UID: \"01215889-00f7-4cfd-97cd-dee00355f9b9\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"
Oct 30 12:40:54.580879 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.580853    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01215889-00f7-4cfd-97cd-dee00355f9b9-tmp\") pod \"kube-auth-proxy-dc45784b6-l4s7t\" (UID: \"01215889-00f7-4cfd-97cd-dee00355f9b9\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"
Oct 30 12:40:54.595926 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.595900    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:54.682060 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.681999    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01215889-00f7-4cfd-97cd-dee00355f9b9-tmp\") pod \"kube-auth-proxy-dc45784b6-l4s7t\" (UID: \"01215889-00f7-4cfd-97cd-dee00355f9b9\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"
Oct 30 12:40:54.682060 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.682040    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78mqb\" (UniqueName: \"kubernetes.io/projected/01215889-00f7-4cfd-97cd-dee00355f9b9-kube-api-access-78mqb\") pod \"kube-auth-proxy-dc45784b6-l4s7t\" (UID: \"01215889-00f7-4cfd-97cd-dee00355f9b9\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"
Oct 30 12:40:54.682213 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.682064    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01215889-00f7-4cfd-97cd-dee00355f9b9-tls-certs\") pod \"kube-auth-proxy-dc45784b6-l4s7t\" (UID: \"01215889-00f7-4cfd-97cd-dee00355f9b9\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"
Oct 30 12:40:54.684155 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.684130    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01215889-00f7-4cfd-97cd-dee00355f9b9-tmp\") pod \"kube-auth-proxy-dc45784b6-l4s7t\" (UID: \"01215889-00f7-4cfd-97cd-dee00355f9b9\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"
Oct 30 12:40:54.684291 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.684274    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01215889-00f7-4cfd-97cd-dee00355f9b9-tls-certs\") pod \"kube-auth-proxy-dc45784b6-l4s7t\" (UID: \"01215889-00f7-4cfd-97cd-dee00355f9b9\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"
Oct 30 12:40:54.702832 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.702812    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78mqb\" (UniqueName: \"kubernetes.io/projected/01215889-00f7-4cfd-97cd-dee00355f9b9-kube-api-access-78mqb\") pod \"kube-auth-proxy-dc45784b6-l4s7t\" (UID: \"01215889-00f7-4cfd-97cd-dee00355f9b9\") " pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"
Oct 30 12:40:54.704159 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.704130    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:54.722892 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.722874    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"
Oct 30 12:40:54.825486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.825454    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:54.925788 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:54.925754    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:55.022921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:55.022892    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t"]
Oct 30 12:40:55.023991 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:40:55.023969    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01215889_00f7_4cfd_97cd_dee00355f9b9.slice/crio-ba37a809b59b2eadf40ee8cf2fa9c03283e73bf75085fcc8900db18a8f373ceb WatchSource:0}: Error finding container ba37a809b59b2eadf40ee8cf2fa9c03283e73bf75085fcc8900db18a8f373ceb: Status 404 returned error can't find the container with id ba37a809b59b2eadf40ee8cf2fa9c03283e73bf75085fcc8900db18a8f373ceb
Oct 30 12:40:55.158600 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:55.158568    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:55.266230 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:55.266199    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:55.456084 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:55.456011    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t" event={"ID":"01215889-00f7-4cfd-97cd-dee00355f9b9","Type":"ContainerStarted","Data":"ba37a809b59b2eadf40ee8cf2fa9c03283e73bf75085fcc8900db18a8f373ceb"}
Oct 30 12:40:55.855822 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:55.855774    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:55.956593 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:55.956550    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:56.569779 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:56.569721    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:57.098449 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.098406    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:40:57.355160 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.355103    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-dc8775675-8b2gn"]
Oct 30 12:40:57.358027 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.358001    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 12:40:57.358464 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.358448    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:40:57.358581 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.358565    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc8775675-8b2gn"
Oct 30 12:40:57.361406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.361389    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"
Oct 30 12:40:57.373434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.373414    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 12:40:57.373561 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.373514    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 12:40:57.377905 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.377884    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dc8775675-8b2gn"]
Oct 30 12:40:57.380954 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.380930    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:40:57.409858 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.409838    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-snqng\""
Oct 30 12:40:57.409964 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.409946    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5twgk\""
Oct 30 12:40:57.416219 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.416196    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 12:40:57.452971 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.452942    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:40:57.464368 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.464327    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t" event={"ID":"01215889-00f7-4cfd-97cd-dee00355f9b9","Type":"ContainerStarted","Data":"1b8a1347f1035f30a8fbbf667b51331081aeeb2315fd7d9082819cd818e179e3"}
Oct 30 12:40:57.501610 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.501557    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-dc45784b6-l4s7t" podStartSLOduration=1.183310776 podStartE2EDuration="3.501543946s" totalImagesPullingTime="2.31823317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:57.500683979 +0000 UTC m=+301.316010460" watchObservedRunningTime="2026-10-30 12:40:57.501543946 +0000 UTC m=+301.316870435"
Oct 30 12:40:57.505570 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.505474    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/71513e79-e472-4ad9-9c03-fd5a83fde281-config-file\") pod \"limitador-limitador-f7d6b5c7c-jbnbk\" (UID: \"71513e79-e472-4ad9-9c03-fd5a83fde281\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"
Oct 30 12:40:57.505724 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.505655    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv7xp\" (UniqueName: \"kubernetes.io/projected/74ada7bd-c6cb-4582-9633-9caeec8757f0-kube-api-access-mv7xp\") pod \"authorino-dc8775675-8b2gn\" (UID: \"74ada7bd-c6cb-4582-9633-9caeec8757f0\") " pod="kuadrant-system/authorino-dc8775675-8b2gn"
Oct 30 12:40:57.505828 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.505810    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr748\" (UniqueName: \"kubernetes.io/projected/71513e79-e472-4ad9-9c03-fd5a83fde281-kube-api-access-fr748\") pod \"limitador-limitador-f7d6b5c7c-jbnbk\" (UID: \"71513e79-e472-4ad9-9c03-fd5a83fde281\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"
Oct 30 12:40:57.606886 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.606802    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/71513e79-e472-4ad9-9c03-fd5a83fde281-config-file\") pod \"limitador-limitador-f7d6b5c7c-jbnbk\" (UID: \"71513e79-e472-4ad9-9c03-fd5a83fde281\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"
Oct 30 12:40:57.606886 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.606862    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv7xp\" (UniqueName: \"kubernetes.io/projected/74ada7bd-c6cb-4582-9633-9caeec8757f0-kube-api-access-mv7xp\") pod \"authorino-dc8775675-8b2gn\" (UID: \"74ada7bd-c6cb-4582-9633-9caeec8757f0\") " pod="kuadrant-system/authorino-dc8775675-8b2gn"
Oct 30 12:40:57.607134 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.606947    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr748\" (UniqueName: \"kubernetes.io/projected/71513e79-e472-4ad9-9c03-fd5a83fde281-kube-api-access-fr748\") pod \"limitador-limitador-f7d6b5c7c-jbnbk\" (UID: \"71513e79-e472-4ad9-9c03-fd5a83fde281\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"
Oct 30 12:40:57.607673 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.607645    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/71513e79-e472-4ad9-9c03-fd5a83fde281-config-file\") pod \"limitador-limitador-f7d6b5c7c-jbnbk\" (UID: \"71513e79-e472-4ad9-9c03-fd5a83fde281\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"
Oct 30 12:40:57.626697 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.626665    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr748\" (UniqueName: \"kubernetes.io/projected/71513e79-e472-4ad9-9c03-fd5a83fde281-kube-api-access-fr748\") pod \"limitador-limitador-f7d6b5c7c-jbnbk\" (UID: \"71513e79-e472-4ad9-9c03-fd5a83fde281\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"
Oct 30 12:40:57.626831 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.626665    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv7xp\" (UniqueName: \"kubernetes.io/projected/74ada7bd-c6cb-4582-9633-9caeec8757f0-kube-api-access-mv7xp\") pod \"authorino-dc8775675-8b2gn\" (UID: \"74ada7bd-c6cb-4582-9633-9caeec8757f0\") " pod="kuadrant-system/authorino-dc8775675-8b2gn"
Oct 30 12:40:57.684317 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.684273    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc8775675-8b2gn"
Oct 30 12:40:57.709537 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.709501    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"
Oct 30 12:40:57.979116 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.979083    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dc8775675-8b2gn"]
Oct 30 12:40:57.980150 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:40:57.980124    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74ada7bd_c6cb_4582_9633_9caeec8757f0.slice/crio-dabdc0076ef458ae04df6290a6bde6da1bce5dbc5bc3064fe184abf422e62919 WatchSource:0}: Error finding container dabdc0076ef458ae04df6290a6bde6da1bce5dbc5bc3064fe184abf422e62919: Status 404 returned error can't find the container with id dabdc0076ef458ae04df6290a6bde6da1bce5dbc5bc3064fe184abf422e62919
Oct 30 12:40:57.981484 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:57.981466    4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:40:58.012501 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:58.012472    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:40:58.013758 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:40:58.013732    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71513e79_e472_4ad9_9c03_fd5a83fde281.slice/crio-4902af04801521500d8e86309208cc67bfe38d8e01fe118d900d71fbbedb88fc WatchSource:0}: Error finding container 4902af04801521500d8e86309208cc67bfe38d8e01fe118d900d71fbbedb88fc: Status 404 returned error can't find the container with id 4902af04801521500d8e86309208cc67bfe38d8e01fe118d900d71fbbedb88fc
Oct 30 12:40:58.469039 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:58.469002    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk" event={"ID":"71513e79-e472-4ad9-9c03-fd5a83fde281","Type":"ContainerStarted","Data":"4902af04801521500d8e86309208cc67bfe38d8e01fe118d900d71fbbedb88fc"}
Oct 30 12:40:58.470194 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:40:58.470137    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc8775675-8b2gn" event={"ID":"74ada7bd-c6cb-4582-9633-9caeec8757f0","Type":"ContainerStarted","Data":"dabdc0076ef458ae04df6290a6bde6da1bce5dbc5bc3064fe184abf422e62919"}
Oct 30 12:41:00.483849 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:00.483773    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk" event={"ID":"71513e79-e472-4ad9-9c03-fd5a83fde281","Type":"ContainerStarted","Data":"839946fa7addc3ad426d6fe8d9624fc45ee7e385e744f4900efc4d64ed0cf829"}
Oct 30 12:41:00.484211 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:00.483882    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"
Oct 30 12:41:00.521703 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:00.521641    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk" podStartSLOduration=1.404874554 podStartE2EDuration="3.521625815s" totalImagesPullingTime="2.116751261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:00.520331131 +0000 UTC m=+304.335657619" watchObservedRunningTime="2026-10-30 12:41:00.521625815 +0000 UTC m=+304.336952309"
Oct 30 12:41:06.509165 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:06.509131    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc8775675-8b2gn" event={"ID":"74ada7bd-c6cb-4582-9633-9caeec8757f0","Type":"ContainerStarted","Data":"f6a578998b502fadc3ec2d231032446eabf378d5eee7b0baa4ab1c6fb8ec2af6"}
Oct 30 12:41:06.547988 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:06.547482    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-dc8775675-8b2gn" podStartSLOduration=1.116642108 podStartE2EDuration="9.54746608s" totalImagesPullingTime="8.430823972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:06.546391193 +0000 UTC m=+310.361717700" watchObservedRunningTime="2026-10-30 12:41:06.54746608 +0000 UTC m=+310.362792571"
Oct 30 12:41:08.347748 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:08.347706    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:08.433422 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:08.433374    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:11.488651 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:11.488621    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"
Oct 30 12:41:16.255751 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:16.255705    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:16.357900 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:16.357872    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:17.068918 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:17.068870    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:21.134804 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:41:21.134766    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:41:36.224991 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:36.224952    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:36.325924 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:36.325888    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:36.440060 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:36.440028    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:36.518125 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:36.518037    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:40.185383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:40.185319    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:40.290047 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:40.290008    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:41.517798 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:41.517758    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:41.648588 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:41.648539    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:42.559746 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:42.559709    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:46.411261 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:46.411228    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:51.132611 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:41:51.132573    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:41:55.565109 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.565076    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"]
Oct 30 12:41:55.568396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.568374    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.595865 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.595842    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"]
Oct 30 12:41:55.643281 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.643257    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj658\" (UniqueName: \"kubernetes.io/projected/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-kube-api-access-rj658\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.643394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.643294    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.643394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.643313    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.643394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.643389    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.643502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.643440    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.643538 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.643522    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.643571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.643550    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.744337 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.744315    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj658\" (UniqueName: \"kubernetes.io/projected/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-kube-api-access-rj658\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.744460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.744363    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.744460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.744389    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.744460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.744409    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.744460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.744444    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.744676 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.744470    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.744676 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.744504    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.745262 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.745240    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.746827 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.746801    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.747039 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.747018    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.747140 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.747124    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.747177 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.747135    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.767924 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.767902    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj658\" (UniqueName: \"kubernetes.io/projected/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-kube-api-access-rj658\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.769478 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.769454    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c7a319-32ec-4793-8d0a-2a0b60fb8d53-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kfdb7\" (UID: \"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:55.880004 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:55.879923    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:56.190070 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:56.190043    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"]
Oct 30 12:41:56.190682 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:41:56.190661    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c7a319_32ec_4793_8d0a_2a0b60fb8d53.slice/crio-c589dcd07b1887605448bb8444610734f4fbd9b5b52e3cb3851b3a11dde27f61 WatchSource:0}: Error finding container c589dcd07b1887605448bb8444610734f4fbd9b5b52e3cb3851b3a11dde27f61: Status 404 returned error can't find the container with id c589dcd07b1887605448bb8444610734f4fbd9b5b52e3cb3851b3a11dde27f61
Oct 30 12:41:56.192668 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:56.192631    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:41:56.192747 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:56.192708    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:41:56.711553 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:56.711521    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7" event={"ID":"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53","Type":"ContainerStarted","Data":"9bcb60f3623b82132a4b45b59366f002138c829ae1c68a469d13916781867e69"}
Oct 30 12:41:56.711941 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:56.711593    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7" event={"ID":"c2c7a319-32ec-4793-8d0a-2a0b60fb8d53","Type":"ContainerStarted","Data":"c589dcd07b1887605448bb8444610734f4fbd9b5b52e3cb3851b3a11dde27f61"}
Oct 30 12:41:56.711941 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:56.711651    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:41:56.729929 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:56.729889    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:56.772881 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:56.772835    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7" podStartSLOduration=1.772824293 podStartE2EDuration="1.772824293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:56.772725085 +0000 UTC m=+360.588051575" watchObservedRunningTime="2026-10-30 12:41:56.772824293 +0000 UTC m=+360.588150781"
Oct 30 12:41:56.829721 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:56.829691    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:56.934023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:56.933988    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:57.030122 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:57.030085    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:57.159160 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:57.159129    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:57.246049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:57.246014    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:57.507550 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:57.507484    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:57.716582 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:57.716549    4522 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kfdb7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:41:57.716919 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:57.716608    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7" podUID="c2c7a319-32ec-4793-8d0a-2a0b60fb8d53" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:41:59.354136 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:59.354106    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:59.454545 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:59.454515    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:59.558591 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:59.558564    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:59.658167 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:59.658110    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:59.756626 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:59.756601    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:41:59.873065 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:41:59.873040    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:00.716581 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:00.716550    4522 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kfdb7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:42:00.716954 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:00.716611    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7" podUID="c2c7a319-32ec-4793-8d0a-2a0b60fb8d53" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:42:03.715996 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:03.715955    4522 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kfdb7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:42:03.716401 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:03.716042    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7" podUID="c2c7a319-32ec-4793-8d0a-2a0b60fb8d53" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:42:06.716207 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:06.716169    4522 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kfdb7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:42:06.716603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:06.716240    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7" podUID="c2c7a319-32ec-4793-8d0a-2a0b60fb8d53" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:42:09.716406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:09.716349    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kfdb7"
Oct 30 12:42:09.801101 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:09.801069    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"]
Oct 30 12:42:09.801277 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:09.801256    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt" podUID="cfb92ec6-6839-4333-a661-e7d8d305cf29" containerName="discovery" containerID="cri-o://2be29610aba89410501a78d3d6adadd485b5054ca25787255969f8ea2cb7b4fa" gracePeriod=30
Oct 30 12:42:10.042100 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.042073    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:42:10.155732 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.155708    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-kube-api-access-c5pvq\" (UniqueName: \"kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-kube-api-access-c5pvq\") pod \"cfb92ec6-6839-4333-a661-e7d8d305cf29\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") "
Oct 30 12:42:10.155866 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.155751    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-cacerts\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-cacerts\") pod \"cfb92ec6-6839-4333-a661-e7d8d305cf29\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") "
Oct 30 12:42:10.155866 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.155772    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-dns-cert\") pod \"cfb92ec6-6839-4333-a661-e7d8d305cf29\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") "
Oct 30 12:42:10.155866 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.155855    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-kubeconfig\") pod \"cfb92ec6-6839-4333-a661-e7d8d305cf29\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") "
Oct 30 12:42:10.156002 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.155888    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-ca-configmap\") pod \"cfb92ec6-6839-4333-a661-e7d8d305cf29\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") "
Oct 30 12:42:10.156002 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.155935    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-token\" (UniqueName: \"kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-token\") pod \"cfb92ec6-6839-4333-a661-e7d8d305cf29\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") "
Oct 30 12:42:10.156002 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.155982    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb92ec6-6839-4333-a661-e7d8d305cf29-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb92ec6-6839-4333-a661-e7d8d305cf29-local-certs\") pod \"cfb92ec6-6839-4333-a661-e7d8d305cf29\" (UID: \"cfb92ec6-6839-4333-a661-e7d8d305cf29\") "
Oct 30 12:42:10.156343 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.156300    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-ca-configmap" pod "cfb92ec6-6839-4333-a661-e7d8d305cf29" (UID: "cfb92ec6-6839-4333-a661-e7d8d305cf29"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:42:10.158127 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.158097    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-cacerts" pod "cfb92ec6-6839-4333-a661-e7d8d305cf29" (UID: "cfb92ec6-6839-4333-a661-e7d8d305cf29"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:42:10.158231 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.158141    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-dns-cert" pod "cfb92ec6-6839-4333-a661-e7d8d305cf29" (UID: "cfb92ec6-6839-4333-a661-e7d8d305cf29"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:42:10.158290 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.158250    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-kube-api-access-c5pvq" pod "cfb92ec6-6839-4333-a661-e7d8d305cf29" (UID: "cfb92ec6-6839-4333-a661-e7d8d305cf29"). InnerVolumeSpecName "kube-api-access-c5pvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:42:10.158290 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.158266    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-kubeconfig" pod "cfb92ec6-6839-4333-a661-e7d8d305cf29" (UID: "cfb92ec6-6839-4333-a661-e7d8d305cf29"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:42:10.162844 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.158366    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-token" pod "cfb92ec6-6839-4333-a661-e7d8d305cf29" (UID: "cfb92ec6-6839-4333-a661-e7d8d305cf29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:42:10.162844 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.159497    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb92ec6-6839-4333-a661-e7d8d305cf29-local-certs" pod "cfb92ec6-6839-4333-a661-e7d8d305cf29" (UID: "cfb92ec6-6839-4333-a661-e7d8d305cf29"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:42:10.256531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.256485    4522 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-kubeconfig\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:10.256531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.256507    4522 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-ca-configmap\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:10.256531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.256518    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:10.256531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.256526    4522 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb92ec6-6839-4333-a661-e7d8d305cf29-local-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:10.256705 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.256534    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5pvq\" (UniqueName: \"kubernetes.io/projected/cfb92ec6-6839-4333-a661-e7d8d305cf29-kube-api-access-c5pvq\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:10.256705 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.256543    4522 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-cacerts\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:10.256705 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.256551    4522 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cfb92ec6-6839-4333-a661-e7d8d305cf29-istio-csr-dns-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:10.764094 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.764063    4522 generic.go:356] "Generic (PLEG): container finished" podID="cfb92ec6-6839-4333-a661-e7d8d305cf29" containerID="2be29610aba89410501a78d3d6adadd485b5054ca25787255969f8ea2cb7b4fa" exitCode=0
Oct 30 12:42:10.764466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.764103    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt" event={"ID":"cfb92ec6-6839-4333-a661-e7d8d305cf29","Type":"ContainerDied","Data":"2be29610aba89410501a78d3d6adadd485b5054ca25787255969f8ea2cb7b4fa"}
Oct 30 12:42:10.764466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.764120    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"
Oct 30 12:42:10.764466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.764139    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt" event={"ID":"cfb92ec6-6839-4333-a661-e7d8d305cf29","Type":"ContainerDied","Data":"1e48f8199c410453024fc256884ab8618c84b8661a799719e621578d1bfccba6"}
Oct 30 12:42:10.764466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.764158    4522 scope.go:122] "RemoveContainer" containerID="2be29610aba89410501a78d3d6adadd485b5054ca25787255969f8ea2cb7b4fa"
Oct 30 12:42:10.772340 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.772321    4522 scope.go:122] "RemoveContainer" containerID="2be29610aba89410501a78d3d6adadd485b5054ca25787255969f8ea2cb7b4fa"
Oct 30 12:42:10.772606 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:42:10.772584    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be29610aba89410501a78d3d6adadd485b5054ca25787255969f8ea2cb7b4fa\": container with ID starting with 2be29610aba89410501a78d3d6adadd485b5054ca25787255969f8ea2cb7b4fa not found: ID does not exist" containerID="2be29610aba89410501a78d3d6adadd485b5054ca25787255969f8ea2cb7b4fa"
Oct 30 12:42:10.772668 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.772618    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be29610aba89410501a78d3d6adadd485b5054ca25787255969f8ea2cb7b4fa"} err="failed to get container status \"2be29610aba89410501a78d3d6adadd485b5054ca25787255969f8ea2cb7b4fa\": rpc error: code = NotFound desc = could not find container \"2be29610aba89410501a78d3d6adadd485b5054ca25787255969f8ea2cb7b4fa\": container with ID starting with 2be29610aba89410501a78d3d6adadd485b5054ca25787255969f8ea2cb7b4fa not found: ID does not exist"
Oct 30 12:42:10.824751 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.824727    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"]
Oct 30 12:42:10.830665 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:10.830642    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b7nrt"]
Oct 30 12:42:11.068111 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:11.068089    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:11.170483 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:11.170453    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:12.331322 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.331292    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb92ec6-6839-4333-a661-e7d8d305cf29" path="/var/lib/kubelet/pods/cfb92ec6-6839-4333-a661-e7d8d305cf29/volumes"
Oct 30 12:42:12.482410 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.482374    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:12.581185 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.581160    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:12.678738 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.678673    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:12.724115 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.724089    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"]
Oct 30 12:42:12.724392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.724379    4522 cpu_manager.go:414] "containerMap: removing container" podUID="cfb92ec6-6839-4333-a661-e7d8d305cf29" containerName="discovery"
Oct 30 12:42:12.724445 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.724393    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb92ec6-6839-4333-a661-e7d8d305cf29" containerName="discovery"
Oct 30 12:42:12.724493 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.724449    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb92ec6-6839-4333-a661-e7d8d305cf29" containerName="discovery"
Oct 30 12:42:12.728557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.728542    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:12.759714 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.759558    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 12:42:12.759714 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.759565    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:42:12.759714 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.759592    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 12:42:12.759714 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.759637    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:42:12.761010 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.760993    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"]
Oct 30 12:42:12.873229 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.873201    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-metrics-certs\") pod \"maas-api-6c8649c4df-njx58\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") " pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:12.873371 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.873235    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-maas-api-tls\") pod \"maas-api-6c8649c4df-njx58\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") " pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:12.873371 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.873270    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-openshift-service-ca\") pod \"maas-api-6c8649c4df-njx58\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") " pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:12.873449 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.873394    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7zn\" (UniqueName: \"kubernetes.io/projected/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-kube-api-access-kf7zn\") pod \"maas-api-6c8649c4df-njx58\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") " pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:12.973841 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.973783    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-metrics-certs\") pod \"maas-api-6c8649c4df-njx58\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") " pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:12.973841 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.973810    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-maas-api-tls\") pod \"maas-api-6c8649c4df-njx58\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") " pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:12.973841 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.973832    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-openshift-service-ca\") pod \"maas-api-6c8649c4df-njx58\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") " pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:12.974016 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.973882    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7zn\" (UniqueName: \"kubernetes.io/projected/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-kube-api-access-kf7zn\") pod \"maas-api-6c8649c4df-njx58\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") " pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:12.974562 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.974534    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-openshift-service-ca\") pod \"maas-api-6c8649c4df-njx58\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") " pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:12.976084 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.976066    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-metrics-certs\") pod \"maas-api-6c8649c4df-njx58\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") " pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:12.976209 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.976150    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-maas-api-tls\") pod \"maas-api-6c8649c4df-njx58\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") " pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:12.992449 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:12.992424    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7zn\" (UniqueName: \"kubernetes.io/projected/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-kube-api-access-kf7zn\") pod \"maas-api-6c8649c4df-njx58\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") " pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:13.041086 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:13.041059    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:13.336822 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:13.336795    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"]
Oct 30 12:42:13.337941 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:42:13.337919    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ad0899_35f0_4c26_b2b5_8ec6c8b522f0.slice/crio-f04fd0a41bab7a844525b128eb91aaea46aee26778e01afded1fcdb31961cf26 WatchSource:0}: Error finding container f04fd0a41bab7a844525b128eb91aaea46aee26778e01afded1fcdb31961cf26: Status 404 returned error can't find the container with id f04fd0a41bab7a844525b128eb91aaea46aee26778e01afded1fcdb31961cf26
Oct 30 12:42:13.779414 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:13.779378    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58" event={"ID":"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0","Type":"ContainerStarted","Data":"f04fd0a41bab7a844525b128eb91aaea46aee26778e01afded1fcdb31961cf26"}
Oct 30 12:42:13.898903 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:13.898862    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:42:15.788895 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:15.788864    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58" event={"ID":"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0","Type":"ContainerStarted","Data":"bbcb350905efcf1809257259c86bab36b0468d10972b93e00ab0a8972154dbb4"}
Oct 30 12:42:15.789198 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:15.788999    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:15.835827 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:15.835778    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58" podStartSLOduration=1.4551805469999999 podStartE2EDuration="3.83576438s" totalImagesPullingTime="2.380583833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:15.834998051 +0000 UTC m=+379.650324541" watchObservedRunningTime="2026-10-30 12:42:15.83576438 +0000 UTC m=+379.651090900"
Oct 30 12:42:21.132865 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:42:21.132831    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:42:21.798538 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:21.798507    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:33.017597 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.017563    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:33.115792 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.115761    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:33.565285 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.565254    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65db9f99d7-l7q4s"]
Oct 30 12:42:33.568598 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.568578    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65db9f99d7-l7q4s"
Oct 30 12:42:33.586221 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.586201    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65db9f99d7-l7q4s"]
Oct 30 12:42:33.597154 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.597134    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 12:42:33.640843 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.640814    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qljg8\" (UniqueName: \"kubernetes.io/projected/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-kube-api-access-qljg8\") pod \"authorino-65db9f99d7-l7q4s\" (UID: \"33ea8df7-ee1a-47ad-ae47-c8d7ff3af554\") " pod="kuadrant-system/authorino-65db9f99d7-l7q4s"
Oct 30 12:42:33.640967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.640853    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-tls-cert\") pod \"authorino-65db9f99d7-l7q4s\" (UID: \"33ea8df7-ee1a-47ad-ae47-c8d7ff3af554\") " pod="kuadrant-system/authorino-65db9f99d7-l7q4s"
Oct 30 12:42:33.741396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.741345    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qljg8\" (UniqueName: \"kubernetes.io/projected/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-kube-api-access-qljg8\") pod \"authorino-65db9f99d7-l7q4s\" (UID: \"33ea8df7-ee1a-47ad-ae47-c8d7ff3af554\") " pod="kuadrant-system/authorino-65db9f99d7-l7q4s"
Oct 30 12:42:33.741521 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.741399    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-tls-cert\") pod \"authorino-65db9f99d7-l7q4s\" (UID: \"33ea8df7-ee1a-47ad-ae47-c8d7ff3af554\") " pod="kuadrant-system/authorino-65db9f99d7-l7q4s"
Oct 30 12:42:33.743712 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.743686    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-tls-cert\") pod \"authorino-65db9f99d7-l7q4s\" (UID: \"33ea8df7-ee1a-47ad-ae47-c8d7ff3af554\") " pod="kuadrant-system/authorino-65db9f99d7-l7q4s"
Oct 30 12:42:33.756897 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.756876    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65db9f99d7-l7q4s"]
Oct 30 12:42:33.757032 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:42:33.757015    4522 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qljg8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-65db9f99d7-l7q4s" podUID="33ea8df7-ee1a-47ad-ae47-c8d7ff3af554"
Oct 30 12:42:33.760614 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.760592    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qljg8\" (UniqueName: \"kubernetes.io/projected/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-kube-api-access-qljg8\") pod \"authorino-65db9f99d7-l7q4s\" (UID: \"33ea8df7-ee1a-47ad-ae47-c8d7ff3af554\") " pod="kuadrant-system/authorino-65db9f99d7-l7q4s"
Oct 30 12:42:33.857422 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.857371    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65db9f99d7-l7q4s"
Oct 30 12:42:33.863833 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.863810    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65db9f99d7-l7q4s"
Oct 30 12:42:33.943297 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.943275    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-tls-cert\" (UniqueName: \"kubernetes.io/secret/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-tls-cert\") pod \"33ea8df7-ee1a-47ad-ae47-c8d7ff3af554\" (UID: \"33ea8df7-ee1a-47ad-ae47-c8d7ff3af554\") "
Oct 30 12:42:33.943406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.943320    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-kube-api-access-qljg8\" (UniqueName: \"kubernetes.io/projected/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-kube-api-access-qljg8\") pod \"33ea8df7-ee1a-47ad-ae47-c8d7ff3af554\" (UID: \"33ea8df7-ee1a-47ad-ae47-c8d7ff3af554\") "
Oct 30 12:42:33.945213 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.945187    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-kube-api-access-qljg8" pod "33ea8df7-ee1a-47ad-ae47-c8d7ff3af554" (UID: "33ea8df7-ee1a-47ad-ae47-c8d7ff3af554"). InnerVolumeSpecName "kube-api-access-qljg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:42:33.945290 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:33.945236    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-tls-cert" pod "33ea8df7-ee1a-47ad-ae47-c8d7ff3af554" (UID: "33ea8df7-ee1a-47ad-ae47-c8d7ff3af554"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:42:34.044685 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.044662    4522 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-tls-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:34.044685 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.044683    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qljg8\" (UniqueName: \"kubernetes.io/projected/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554-kube-api-access-qljg8\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:34.570971 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.570946    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8fd5b54-cwv87"]
Oct 30 12:42:34.574081 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.574064    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8fd5b54-cwv87"
Oct 30 12:42:34.591246 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.591197    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8fd5b54-cwv87"]
Oct 30 12:42:34.649406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.649382    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-tls-cert\") pod \"authorino-8fd5b54-cwv87\" (UID: \"d7d9458e-e3cf-47df-a993-cdc9a4e8d768\") " pod="kuadrant-system/authorino-8fd5b54-cwv87"
Oct 30 12:42:34.649499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.649410    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8blz\" (UniqueName: \"kubernetes.io/projected/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-kube-api-access-v8blz\") pod \"authorino-8fd5b54-cwv87\" (UID: \"d7d9458e-e3cf-47df-a993-cdc9a4e8d768\") " pod="kuadrant-system/authorino-8fd5b54-cwv87"
Oct 30 12:42:34.749942 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.749913    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-tls-cert\") pod \"authorino-8fd5b54-cwv87\" (UID: \"d7d9458e-e3cf-47df-a993-cdc9a4e8d768\") " pod="kuadrant-system/authorino-8fd5b54-cwv87"
Oct 30 12:42:34.749942 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.749942    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8blz\" (UniqueName: \"kubernetes.io/projected/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-kube-api-access-v8blz\") pod \"authorino-8fd5b54-cwv87\" (UID: \"d7d9458e-e3cf-47df-a993-cdc9a4e8d768\") " pod="kuadrant-system/authorino-8fd5b54-cwv87"
Oct 30 12:42:34.752218 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.752195    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-tls-cert\") pod \"authorino-8fd5b54-cwv87\" (UID: \"d7d9458e-e3cf-47df-a993-cdc9a4e8d768\") " pod="kuadrant-system/authorino-8fd5b54-cwv87"
Oct 30 12:42:34.767942 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.767924    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8blz\" (UniqueName: \"kubernetes.io/projected/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-kube-api-access-v8blz\") pod \"authorino-8fd5b54-cwv87\" (UID: \"d7d9458e-e3cf-47df-a993-cdc9a4e8d768\") " pod="kuadrant-system/authorino-8fd5b54-cwv87"
Oct 30 12:42:34.861240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.861194    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65db9f99d7-l7q4s"
Oct 30 12:42:34.885070 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.885044    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8fd5b54-cwv87"
Oct 30 12:42:34.937038 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.936853    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65db9f99d7-l7q4s"]
Oct 30 12:42:34.947322 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:34.941805    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65db9f99d7-l7q4s"]
Oct 30 12:42:35.180273 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:35.180252    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8fd5b54-cwv87"]
Oct 30 12:42:35.181486 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:42:35.181460    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7d9458e_e3cf_47df_a993_cdc9a4e8d768.slice/crio-0b1791569404749afd53c2a5cd5b72ccf577302e08bbace7ac06905c624a4e50 WatchSource:0}: Error finding container 0b1791569404749afd53c2a5cd5b72ccf577302e08bbace7ac06905c624a4e50: Status 404 returned error can't find the container with id 0b1791569404749afd53c2a5cd5b72ccf577302e08bbace7ac06905c624a4e50
Oct 30 12:42:35.866753 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:35.866650    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8fd5b54-cwv87" event={"ID":"d7d9458e-e3cf-47df-a993-cdc9a4e8d768","Type":"ContainerStarted","Data":"ae97b64af26b7c2a68e1004b543d126b6a64c203cee942122440db6407b4c6ae"}
Oct 30 12:42:35.866753 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:35.866756    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8fd5b54-cwv87" event={"ID":"d7d9458e-e3cf-47df-a993-cdc9a4e8d768","Type":"ContainerStarted","Data":"0b1791569404749afd53c2a5cd5b72ccf577302e08bbace7ac06905c624a4e50"}
Oct 30 12:42:35.912603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:35.912551    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-8fd5b54-cwv87" podStartSLOduration=1.349448988 podStartE2EDuration="1.912535227s" totalImagesPullingTime="563.086239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:35.911177164 +0000 UTC m=+399.726503654" watchObservedRunningTime="2026-10-30 12:42:35.912535227 +0000 UTC m=+399.727861727"
Oct 30 12:42:36.039395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.039343    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-dc8775675-8b2gn"]
Oct 30 12:42:36.039583 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.039556    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-dc8775675-8b2gn" podUID="74ada7bd-c6cb-4582-9633-9caeec8757f0" containerName="authorino" containerID="cri-o://f6a578998b502fadc3ec2d231032446eabf378d5eee7b0baa4ab1c6fb8ec2af6" gracePeriod=30
Oct 30 12:42:36.291056 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.291002    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc8775675-8b2gn"
Oct 30 12:42:36.337726 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.337693    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ea8df7-ee1a-47ad-ae47-c8d7ff3af554" path="/var/lib/kubelet/pods/33ea8df7-ee1a-47ad-ae47-c8d7ff3af554/volumes"
Oct 30 12:42:36.369245 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.369194    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ada7bd-c6cb-4582-9633-9caeec8757f0-kube-api-access-mv7xp\" (UniqueName: \"kubernetes.io/projected/74ada7bd-c6cb-4582-9633-9caeec8757f0-kube-api-access-mv7xp\") pod \"74ada7bd-c6cb-4582-9633-9caeec8757f0\" (UID: \"74ada7bd-c6cb-4582-9633-9caeec8757f0\") "
Oct 30 12:42:36.371598 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.371571    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ada7bd-c6cb-4582-9633-9caeec8757f0-kube-api-access-mv7xp" pod "74ada7bd-c6cb-4582-9633-9caeec8757f0" (UID: "74ada7bd-c6cb-4582-9633-9caeec8757f0"). InnerVolumeSpecName "kube-api-access-mv7xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:42:36.470001 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.469968    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv7xp\" (UniqueName: \"kubernetes.io/projected/74ada7bd-c6cb-4582-9633-9caeec8757f0-kube-api-access-mv7xp\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:36.871045 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.871012    4522 generic.go:356] "Generic (PLEG): container finished" podID="74ada7bd-c6cb-4582-9633-9caeec8757f0" containerID="f6a578998b502fadc3ec2d231032446eabf378d5eee7b0baa4ab1c6fb8ec2af6" exitCode=0
Oct 30 12:42:36.871183 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.871054    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc8775675-8b2gn"
Oct 30 12:42:36.871183 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.871096    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc8775675-8b2gn" event={"ID":"74ada7bd-c6cb-4582-9633-9caeec8757f0","Type":"ContainerDied","Data":"f6a578998b502fadc3ec2d231032446eabf378d5eee7b0baa4ab1c6fb8ec2af6"}
Oct 30 12:42:36.871183 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.871137    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc8775675-8b2gn" event={"ID":"74ada7bd-c6cb-4582-9633-9caeec8757f0","Type":"ContainerDied","Data":"dabdc0076ef458ae04df6290a6bde6da1bce5dbc5bc3064fe184abf422e62919"}
Oct 30 12:42:36.871183 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.871153    4522 scope.go:122] "RemoveContainer" containerID="f6a578998b502fadc3ec2d231032446eabf378d5eee7b0baa4ab1c6fb8ec2af6"
Oct 30 12:42:36.879672 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.879656    4522 scope.go:122] "RemoveContainer" containerID="f6a578998b502fadc3ec2d231032446eabf378d5eee7b0baa4ab1c6fb8ec2af6"
Oct 30 12:42:36.879908 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:42:36.879890    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a578998b502fadc3ec2d231032446eabf378d5eee7b0baa4ab1c6fb8ec2af6\": container with ID starting with f6a578998b502fadc3ec2d231032446eabf378d5eee7b0baa4ab1c6fb8ec2af6 not found: ID does not exist" containerID="f6a578998b502fadc3ec2d231032446eabf378d5eee7b0baa4ab1c6fb8ec2af6"
Oct 30 12:42:36.879964 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.879921    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a578998b502fadc3ec2d231032446eabf378d5eee7b0baa4ab1c6fb8ec2af6"} err="failed to get container status \"f6a578998b502fadc3ec2d231032446eabf378d5eee7b0baa4ab1c6fb8ec2af6\": rpc error: code = NotFound desc = could not find container \"f6a578998b502fadc3ec2d231032446eabf378d5eee7b0baa4ab1c6fb8ec2af6\": container with ID starting with f6a578998b502fadc3ec2d231032446eabf378d5eee7b0baa4ab1c6fb8ec2af6 not found: ID does not exist"
Oct 30 12:42:36.919034 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.919004    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-dc8775675-8b2gn"]
Oct 30 12:42:36.922811 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:36.922792    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-dc8775675-8b2gn"]
Oct 30 12:42:38.332714 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:38.332683    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ada7bd-c6cb-4582-9633-9caeec8757f0" path="/var/lib/kubelet/pods/74ada7bd-c6cb-4582-9633-9caeec8757f0/volumes"
Oct 30 12:42:42.843422 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:42.843380    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"]
Oct 30 12:42:42.843962 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:42.843626    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58" podUID="c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0" containerName="maas-api" containerID="cri-o://bbcb350905efcf1809257259c86bab36b0468d10972b93e00ab0a8972154dbb4" gracePeriod=30
Oct 30 12:42:43.081805 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.081781    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:43.118235 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.118114    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-metrics-certs\") pod \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") "
Oct 30 12:42:43.118235 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.118170    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-openshift-service-ca\") pod \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") "
Oct 30 12:42:43.118417 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.118275    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-kube-api-access-kf7zn\" (UniqueName: \"kubernetes.io/projected/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-kube-api-access-kf7zn\") pod \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") "
Oct 30 12:42:43.118417 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.118387    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-maas-api-tls\") pod \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\" (UID: \"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0\") "
Oct 30 12:42:43.118543 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.118517    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-openshift-service-ca" pod "c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0" (UID: "c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:42:43.118639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.118611    4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-openshift-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:43.120693 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.120669    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-metrics-certs" pod "c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0" (UID: "c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:42:43.120850 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.120817    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-maas-api-tls" pod "c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0" (UID: "c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:42:43.121197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.121158    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-kube-api-access-kf7zn" pod "c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0" (UID: "c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0"). InnerVolumeSpecName "kube-api-access-kf7zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:42:43.219420 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.219391    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:43.219516 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.219421    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf7zn\" (UniqueName: \"kubernetes.io/projected/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-kube-api-access-kf7zn\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:43.219516 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.219436    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0-maas-api-tls\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:42:43.900423 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.900384    4522 generic.go:356] "Generic (PLEG): container finished" podID="c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0" containerID="bbcb350905efcf1809257259c86bab36b0468d10972b93e00ab0a8972154dbb4" exitCode=0
Oct 30 12:42:43.900844 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.900433    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"
Oct 30 12:42:43.900844 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.900444    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58" event={"ID":"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0","Type":"ContainerDied","Data":"bbcb350905efcf1809257259c86bab36b0468d10972b93e00ab0a8972154dbb4"}
Oct 30 12:42:43.900844 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.900486    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c8649c4df-njx58" event={"ID":"c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0","Type":"ContainerDied","Data":"f04fd0a41bab7a844525b128eb91aaea46aee26778e01afded1fcdb31961cf26"}
Oct 30 12:42:43.900844 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.900508    4522 scope.go:122] "RemoveContainer" containerID="bbcb350905efcf1809257259c86bab36b0468d10972b93e00ab0a8972154dbb4"
Oct 30 12:42:43.909402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.909385    4522 scope.go:122] "RemoveContainer" containerID="bbcb350905efcf1809257259c86bab36b0468d10972b93e00ab0a8972154dbb4"
Oct 30 12:42:43.909654 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:42:43.909636    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbcb350905efcf1809257259c86bab36b0468d10972b93e00ab0a8972154dbb4\": container with ID starting with bbcb350905efcf1809257259c86bab36b0468d10972b93e00ab0a8972154dbb4 not found: ID does not exist" containerID="bbcb350905efcf1809257259c86bab36b0468d10972b93e00ab0a8972154dbb4"
Oct 30 12:42:43.909702 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.909661    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbcb350905efcf1809257259c86bab36b0468d10972b93e00ab0a8972154dbb4"} err="failed to get container status \"bbcb350905efcf1809257259c86bab36b0468d10972b93e00ab0a8972154dbb4\": rpc error: code = NotFound desc = could not find container \"bbcb350905efcf1809257259c86bab36b0468d10972b93e00ab0a8972154dbb4\": container with ID starting with bbcb350905efcf1809257259c86bab36b0468d10972b93e00ab0a8972154dbb4 not found: ID does not exist"
Oct 30 12:42:43.949114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.949088    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"]
Oct 30 12:42:43.960924 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:43.960897    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c8649c4df-njx58"]
Oct 30 12:42:44.332162 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:44.332124    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0" path="/var/lib/kubelet/pods/c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0/volumes"
Oct 30 12:42:46.530146 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:42:46.530080    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:42:51.136548 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:42:51.136512    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:43:21.136134 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:43:21.136082    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:43:31.063873 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:31.063836    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:31.151732 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:31.151707    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:39.282812 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:39.282779    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:39.884145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:39.884108    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:41.071647 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.071613    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:41.104621 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.104594    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p8qh9"]
Oct 30 12:43:41.104862 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.104849    4522 cpu_manager.go:414] "containerMap: removing container" podUID="c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0" containerName="maas-api"
Oct 30 12:43:41.104917 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.104863    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0" containerName="maas-api"
Oct 30 12:43:41.104917 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.104886    4522 cpu_manager.go:414] "containerMap: removing container" podUID="74ada7bd-c6cb-4582-9633-9caeec8757f0" containerName="authorino"
Oct 30 12:43:41.104917 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.104891    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ada7bd-c6cb-4582-9633-9caeec8757f0" containerName="authorino"
Oct 30 12:43:41.105032 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.104951    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ad0899-35f0-4c26-b2b5-8ec6c8b522f0" containerName="maas-api"
Oct 30 12:43:41.105032 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.104966    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ada7bd-c6cb-4582-9633-9caeec8757f0" containerName="authorino"
Oct 30 12:43:41.107877 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.107859    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9"
Oct 30 12:43:41.134219 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.134185    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 12:43:41.134219 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.134207    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 12:43:41.134386 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.134347    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 12:43:41.134455 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.134403    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 12:43:41.134455 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.134412    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-25pn5\""
Oct 30 12:43:41.135642 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.135620    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p8qh9"]
Oct 30 12:43:41.171629 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.171592    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:41.229198 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.229174    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtp46\" (UniqueName: \"kubernetes.io/projected/e375bc72-ab6c-4c73-ac41-c31384a0623c-kube-api-access-jtp46\") pod \"ai-gateway-controller-7d85675f98-p8qh9\" (UID: \"e375bc72-ab6c-4c73-ac41-c31384a0623c\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9"
Oct 30 12:43:41.229324 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.229213    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e375bc72-ab6c-4c73-ac41-c31384a0623c-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-p8qh9\" (UID: \"e375bc72-ab6c-4c73-ac41-c31384a0623c\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9"
Oct 30 12:43:41.329850 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.329777    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtp46\" (UniqueName: \"kubernetes.io/projected/e375bc72-ab6c-4c73-ac41-c31384a0623c-kube-api-access-jtp46\") pod \"ai-gateway-controller-7d85675f98-p8qh9\" (UID: \"e375bc72-ab6c-4c73-ac41-c31384a0623c\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9"
Oct 30 12:43:41.329850 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.329818    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e375bc72-ab6c-4c73-ac41-c31384a0623c-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-p8qh9\" (UID: \"e375bc72-ab6c-4c73-ac41-c31384a0623c\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9"
Oct 30 12:43:41.332153 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.332125    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e375bc72-ab6c-4c73-ac41-c31384a0623c-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-p8qh9\" (UID: \"e375bc72-ab6c-4c73-ac41-c31384a0623c\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9"
Oct 30 12:43:41.348987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.348962    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtp46\" (UniqueName: \"kubernetes.io/projected/e375bc72-ab6c-4c73-ac41-c31384a0623c-kube-api-access-jtp46\") pod \"ai-gateway-controller-7d85675f98-p8qh9\" (UID: \"e375bc72-ab6c-4c73-ac41-c31384a0623c\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9"
Oct 30 12:43:41.420784 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.420762    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9"
Oct 30 12:43:41.715062 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:41.715035    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p8qh9"]
Oct 30 12:43:41.716716 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:43:41.716688    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode375bc72_ab6c_4c73_ac41_c31384a0623c.slice/crio-ddcc1a618b9e7f5a3dd7bd5ab61447d4b254064a7f9cd9d385851b57f4319228 WatchSource:0}: Error finding container ddcc1a618b9e7f5a3dd7bd5ab61447d4b254064a7f9cd9d385851b57f4319228: Status 404 returned error can't find the container with id ddcc1a618b9e7f5a3dd7bd5ab61447d4b254064a7f9cd9d385851b57f4319228
Oct 30 12:43:42.125384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:42.125330    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9" event={"ID":"e375bc72-ab6c-4c73-ac41-c31384a0623c","Type":"ContainerStarted","Data":"ddcc1a618b9e7f5a3dd7bd5ab61447d4b254064a7f9cd9d385851b57f4319228"}
Oct 30 12:43:43.029624 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:43.029584    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:43.129348 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:43.129319    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:44.134981 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:44.134943    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9" event={"ID":"e375bc72-ab6c-4c73-ac41-c31384a0623c","Type":"ContainerStarted","Data":"5e0f1fcd56ad2e00ea780a93a46ea0f9b8d68a024cb3fbec28ab3c9928923425"}
Oct 30 12:43:44.135437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:44.135046    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9"
Oct 30 12:43:44.179151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:44.179104    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9" podStartSLOduration=1.756416816 podStartE2EDuration="3.179091982s" totalImagesPullingTime="1.422675166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:44.178031996 +0000 UTC m=+467.993358484" watchObservedRunningTime="2026-10-30 12:43:44.179091982 +0000 UTC m=+467.994418470"
Oct 30 12:43:46.653588 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:46.653553    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:51.136471 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:43:51.136428    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:43:54.180378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:54.180323    4522 generic.go:356] "Generic (PLEG): container finished" podID="e375bc72-ab6c-4c73-ac41-c31384a0623c" containerID="5e0f1fcd56ad2e00ea780a93a46ea0f9b8d68a024cb3fbec28ab3c9928923425" exitCode=1
Oct 30 12:43:54.180848 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:54.180383    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9" event={"ID":"e375bc72-ab6c-4c73-ac41-c31384a0623c","Type":"ContainerDied","Data":"5e0f1fcd56ad2e00ea780a93a46ea0f9b8d68a024cb3fbec28ab3c9928923425"}
Oct 30 12:43:54.180848 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:54.180681    4522 scope.go:122] "RemoveContainer" containerID="5e0f1fcd56ad2e00ea780a93a46ea0f9b8d68a024cb3fbec28ab3c9928923425"
Oct 30 12:43:55.186364 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.186317    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9" event={"ID":"e375bc72-ab6c-4c73-ac41-c31384a0623c","Type":"ContainerStarted","Data":"33ab81c3be7ee3e2d406f0372a8347e7e15663259ef042b0147da6601888a95e"}
Oct 30 12:43:55.186712 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.186542    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9"
Oct 30 12:43:55.617147 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.617118    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:55.715192 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.715164    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:43:55.735978 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.735949    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-hv6zx"]
Oct 30 12:43:55.739083 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.739065    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-hv6zx"
Oct 30 12:43:55.755197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.755179    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rsnkh\""
Oct 30 12:43:55.755197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.755165    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 12:43:55.768280 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.768261    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-hv6zx"]
Oct 30 12:43:55.795255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.795234    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z"]
Oct 30 12:43:55.799230 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.799208    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z"
Oct 30 12:43:55.815930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.815909    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z"]
Oct 30 12:43:55.823235 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.823200    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh26z\" (UniqueName: \"kubernetes.io/projected/f7be2765-5a1f-42db-94fd-91c0349e403d-kube-api-access-bh26z\") pod \"payload-pre-processing-7b98bb678c-pdt6z\" (UID: \"f7be2765-5a1f-42db-94fd-91c0349e403d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z"
Oct 30 12:43:55.823235 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.823231    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4098fe5a-e7d3-488c-a1ed-72d447999e15-plugins-config-volume\") pod \"payload-processing-5f78687d5b-hv6zx\" (UID: \"4098fe5a-e7d3-488c-a1ed-72d447999e15\") " pod="openshift-ingress/payload-processing-5f78687d5b-hv6zx"
Oct 30 12:43:55.823417 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.823255    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9fvp\" (UniqueName: \"kubernetes.io/projected/4098fe5a-e7d3-488c-a1ed-72d447999e15-kube-api-access-x9fvp\") pod \"payload-processing-5f78687d5b-hv6zx\" (UID: \"4098fe5a-e7d3-488c-a1ed-72d447999e15\") " pod="openshift-ingress/payload-processing-5f78687d5b-hv6zx"
Oct 30 12:43:55.823417 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.823298    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7be2765-5a1f-42db-94fd-91c0349e403d-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-pdt6z\" (UID: \"f7be2765-5a1f-42db-94fd-91c0349e403d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z"
Oct 30 12:43:55.923898 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.923836    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh26z\" (UniqueName: \"kubernetes.io/projected/f7be2765-5a1f-42db-94fd-91c0349e403d-kube-api-access-bh26z\") pod \"payload-pre-processing-7b98bb678c-pdt6z\" (UID: \"f7be2765-5a1f-42db-94fd-91c0349e403d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z"
Oct 30 12:43:55.923898 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.923863    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4098fe5a-e7d3-488c-a1ed-72d447999e15-plugins-config-volume\") pod \"payload-processing-5f78687d5b-hv6zx\" (UID: \"4098fe5a-e7d3-488c-a1ed-72d447999e15\") " pod="openshift-ingress/payload-processing-5f78687d5b-hv6zx"
Oct 30 12:43:55.923898 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.923882    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9fvp\" (UniqueName: \"kubernetes.io/projected/4098fe5a-e7d3-488c-a1ed-72d447999e15-kube-api-access-x9fvp\") pod \"payload-processing-5f78687d5b-hv6zx\" (UID: \"4098fe5a-e7d3-488c-a1ed-72d447999e15\") " pod="openshift-ingress/payload-processing-5f78687d5b-hv6zx"
Oct 30 12:43:55.924066 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.923916    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7be2765-5a1f-42db-94fd-91c0349e403d-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-pdt6z\" (UID: \"f7be2765-5a1f-42db-94fd-91c0349e403d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z"
Oct 30 12:43:55.924605 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.924586    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7be2765-5a1f-42db-94fd-91c0349e403d-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-pdt6z\" (UID: \"f7be2765-5a1f-42db-94fd-91c0349e403d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z"
Oct 30 12:43:55.924648 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.924623    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4098fe5a-e7d3-488c-a1ed-72d447999e15-plugins-config-volume\") pod \"payload-processing-5f78687d5b-hv6zx\" (UID: \"4098fe5a-e7d3-488c-a1ed-72d447999e15\") " pod="openshift-ingress/payload-processing-5f78687d5b-hv6zx"
Oct 30 12:43:55.942632 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.942606    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh26z\" (UniqueName: \"kubernetes.io/projected/f7be2765-5a1f-42db-94fd-91c0349e403d-kube-api-access-bh26z\") pod \"payload-pre-processing-7b98bb678c-pdt6z\" (UID: \"f7be2765-5a1f-42db-94fd-91c0349e403d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z"
Oct 30 12:43:55.942716 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:55.942664    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9fvp\" (UniqueName: \"kubernetes.io/projected/4098fe5a-e7d3-488c-a1ed-72d447999e15-kube-api-access-x9fvp\") pod \"payload-processing-5f78687d5b-hv6zx\" (UID: \"4098fe5a-e7d3-488c-a1ed-72d447999e15\") " pod="openshift-ingress/payload-processing-5f78687d5b-hv6zx"
Oct 30 12:43:56.051760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:56.051737    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-hv6zx"
Oct 30 12:43:56.111846 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:56.111815    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z"
Oct 30 12:43:56.342780 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:56.342756    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-hv6zx"]
Oct 30 12:43:56.343163 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:43:56.343141    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4098fe5a_e7d3_488c_a1ed_72d447999e15.slice/crio-57308d544a56879b1f95a2c0309a615d0996920f43894ed04dda43321a85b191 WatchSource:0}: Error finding container 57308d544a56879b1f95a2c0309a615d0996920f43894ed04dda43321a85b191: Status 404 returned error can't find the container with id 57308d544a56879b1f95a2c0309a615d0996920f43894ed04dda43321a85b191
Oct 30 12:43:56.407579 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:56.407559    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z"]
Oct 30 12:43:56.408195 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:43:56.408171    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7be2765_5a1f_42db_94fd_91c0349e403d.slice/crio-69ccdb115bed46e1a5dc9a809c01310f4d6e2e2359ab2cd67771d76b0f843688 WatchSource:0}: Error finding container 69ccdb115bed46e1a5dc9a809c01310f4d6e2e2359ab2cd67771d76b0f843688: Status 404 returned error can't find the container with id 69ccdb115bed46e1a5dc9a809c01310f4d6e2e2359ab2cd67771d76b0f843688
Oct 30 12:43:57.197329 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:57.197287    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-hv6zx" event={"ID":"4098fe5a-e7d3-488c-a1ed-72d447999e15","Type":"ContainerStarted","Data":"57308d544a56879b1f95a2c0309a615d0996920f43894ed04dda43321a85b191"}
Oct 30 12:43:57.198768 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:57.198740    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z" event={"ID":"f7be2765-5a1f-42db-94fd-91c0349e403d","Type":"ContainerStarted","Data":"69ccdb115bed46e1a5dc9a809c01310f4d6e2e2359ab2cd67771d76b0f843688"}
Oct 30 12:43:59.210294 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:59.210252    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z" event={"ID":"f7be2765-5a1f-42db-94fd-91c0349e403d","Type":"ContainerStarted","Data":"1898661609c2a816182f584084d7c56b08808080cdeb3c3609a27986773b3276"}
Oct 30 12:43:59.210759 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:59.210379    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z"
Oct 30 12:43:59.212036 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:59.212011    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-hv6zx" event={"ID":"4098fe5a-e7d3-488c-a1ed-72d447999e15","Type":"ContainerStarted","Data":"b4bffd6e0ea89de0fd45b8e1867770130dd63240af4fbcb186ef218274b6a492"}
Oct 30 12:43:59.212186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:59.212135    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-hv6zx"
Oct 30 12:43:59.248005 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:59.247957    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z" podStartSLOduration=2.185999196 podStartE2EDuration="4.247944714s" totalImagesPullingTime="2.061945518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:59.247660271 +0000 UTC m=+483.062986776" watchObservedRunningTime="2026-10-30 12:43:59.247944714 +0000 UTC m=+483.063271203"
Oct 30 12:43:59.284752 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:43:59.284710    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-hv6zx" podStartSLOduration=2.1254353 podStartE2EDuration="4.28469925s" totalImagesPullingTime="2.15926395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:59.284462304 +0000 UTC m=+483.099788774" watchObservedRunningTime="2026-10-30 12:43:59.28469925 +0000 UTC m=+483.100025738"
Oct 30 12:44:06.193533 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:06.193495    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9"
Oct 30 12:44:06.484965 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:06.484879    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-55f5889bd4-fv926"]
Oct 30 12:44:06.488567 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:06.488540    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55f5889bd4-fv926"
Oct 30 12:44:06.508069 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:06.508040    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55f5889bd4-fv926"]
Oct 30 12:44:06.600516 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:06.600480    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-948q9\" (UniqueName: \"kubernetes.io/projected/0ef93d47-69d0-4e66-9528-009fb281016d-kube-api-access-948q9\") pod \"ai-gateway-controller-55f5889bd4-fv926\" (UID: \"0ef93d47-69d0-4e66-9528-009fb281016d\") " pod="opendatahub/ai-gateway-controller-55f5889bd4-fv926"
Oct 30 12:44:06.600666 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:06.600535    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ef93d47-69d0-4e66-9528-009fb281016d-metrics-certs\") pod \"ai-gateway-controller-55f5889bd4-fv926\" (UID: \"0ef93d47-69d0-4e66-9528-009fb281016d\") " pod="opendatahub/ai-gateway-controller-55f5889bd4-fv926"
Oct 30 12:44:06.701411 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:06.701382    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-948q9\" (UniqueName: \"kubernetes.io/projected/0ef93d47-69d0-4e66-9528-009fb281016d-kube-api-access-948q9\") pod \"ai-gateway-controller-55f5889bd4-fv926\" (UID: \"0ef93d47-69d0-4e66-9528-009fb281016d\") " pod="opendatahub/ai-gateway-controller-55f5889bd4-fv926"
Oct 30 12:44:06.701513 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:06.701424    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ef93d47-69d0-4e66-9528-009fb281016d-metrics-certs\") pod \"ai-gateway-controller-55f5889bd4-fv926\" (UID: \"0ef93d47-69d0-4e66-9528-009fb281016d\") " pod="opendatahub/ai-gateway-controller-55f5889bd4-fv926"
Oct 30 12:44:06.703643 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:06.703619    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ef93d47-69d0-4e66-9528-009fb281016d-metrics-certs\") pod \"ai-gateway-controller-55f5889bd4-fv926\" (UID: \"0ef93d47-69d0-4e66-9528-009fb281016d\") " pod="opendatahub/ai-gateway-controller-55f5889bd4-fv926"
Oct 30 12:44:06.728132 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:06.728108    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-948q9\" (UniqueName: \"kubernetes.io/projected/0ef93d47-69d0-4e66-9528-009fb281016d-kube-api-access-948q9\") pod \"ai-gateway-controller-55f5889bd4-fv926\" (UID: \"0ef93d47-69d0-4e66-9528-009fb281016d\") " pod="opendatahub/ai-gateway-controller-55f5889bd4-fv926"
Oct 30 12:44:06.803450 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:06.803426    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55f5889bd4-fv926"
Oct 30 12:44:07.097637 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:07.097611    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55f5889bd4-fv926"]
Oct 30 12:44:07.098994 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:44:07.098968    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef93d47_69d0_4e66_9528_009fb281016d.slice/crio-e38ea732f9de070d3613222d4e719f68a0e9d8646aad995624f26caafed0035f WatchSource:0}: Error finding container e38ea732f9de070d3613222d4e719f68a0e9d8646aad995624f26caafed0035f: Status 404 returned error can't find the container with id e38ea732f9de070d3613222d4e719f68a0e9d8646aad995624f26caafed0035f
Oct 30 12:44:07.247547 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:07.247517    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55f5889bd4-fv926" event={"ID":"0ef93d47-69d0-4e66-9528-009fb281016d","Type":"ContainerStarted","Data":"e38ea732f9de070d3613222d4e719f68a0e9d8646aad995624f26caafed0035f"}
Oct 30 12:44:08.253115 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:08.253081    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55f5889bd4-fv926" event={"ID":"0ef93d47-69d0-4e66-9528-009fb281016d","Type":"ContainerStarted","Data":"fcd476840f74a229ef61668af91272caf70f9306e1ff090937abb78a5ccc8292"}
Oct 30 12:44:08.253616 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:08.253188    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-55f5889bd4-fv926"
Oct 30 12:44:08.292418 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:08.292371    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-55f5889bd4-fv926" podStartSLOduration=1.9841975139999999 podStartE2EDuration="2.292339118s" totalImagesPullingTime="308.141604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:08.292156844 +0000 UTC m=+492.107483332" watchObservedRunningTime="2026-10-30 12:44:08.292339118 +0000 UTC m=+492.107665607"
Oct 30 12:44:10.219196 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:10.219168    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pdt6z"
Oct 30 12:44:10.219602 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:10.219230    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-hv6zx"
Oct 30 12:44:19.260482 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:19.260446    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-55f5889bd4-fv926"
Oct 30 12:44:19.334885 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:19.334852    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p8qh9"]
Oct 30 12:44:19.335097 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:19.335067    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9" podUID="e375bc72-ab6c-4c73-ac41-c31384a0623c" containerName="manager" containerID="cri-o://33ab81c3be7ee3e2d406f0372a8347e7e15663259ef042b0147da6601888a95e" gracePeriod=30
Oct 30 12:44:19.567306 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:19.567283    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9"
Oct 30 12:44:19.690756 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:19.690731    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e375bc72-ab6c-4c73-ac41-c31384a0623c-kube-api-access-jtp46\" (UniqueName: \"kubernetes.io/projected/e375bc72-ab6c-4c73-ac41-c31384a0623c-kube-api-access-jtp46\") pod \"e375bc72-ab6c-4c73-ac41-c31384a0623c\" (UID: \"e375bc72-ab6c-4c73-ac41-c31384a0623c\") "
Oct 30 12:44:19.690889 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:19.690763    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e375bc72-ab6c-4c73-ac41-c31384a0623c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e375bc72-ab6c-4c73-ac41-c31384a0623c-metrics-certs\") pod \"e375bc72-ab6c-4c73-ac41-c31384a0623c\" (UID: \"e375bc72-ab6c-4c73-ac41-c31384a0623c\") "
Oct 30 12:44:19.692825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:19.692798    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e375bc72-ab6c-4c73-ac41-c31384a0623c-metrics-certs" pod "e375bc72-ab6c-4c73-ac41-c31384a0623c" (UID: "e375bc72-ab6c-4c73-ac41-c31384a0623c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:19.692923 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:19.692892    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e375bc72-ab6c-4c73-ac41-c31384a0623c-kube-api-access-jtp46" pod "e375bc72-ab6c-4c73-ac41-c31384a0623c" (UID: "e375bc72-ab6c-4c73-ac41-c31384a0623c"). InnerVolumeSpecName "kube-api-access-jtp46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:19.791739 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:19.791711    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtp46\" (UniqueName: \"kubernetes.io/projected/e375bc72-ab6c-4c73-ac41-c31384a0623c-kube-api-access-jtp46\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:44:19.791739 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:19.791736    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e375bc72-ab6c-4c73-ac41-c31384a0623c-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:44:20.306854 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:20.306822    4522 generic.go:356] "Generic (PLEG): container finished" podID="e375bc72-ab6c-4c73-ac41-c31384a0623c" containerID="33ab81c3be7ee3e2d406f0372a8347e7e15663259ef042b0147da6601888a95e" exitCode=0
Oct 30 12:44:20.307216 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:20.306871    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9" event={"ID":"e375bc72-ab6c-4c73-ac41-c31384a0623c","Type":"ContainerDied","Data":"33ab81c3be7ee3e2d406f0372a8347e7e15663259ef042b0147da6601888a95e"}
Oct 30 12:44:20.307216 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:20.306918    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9" event={"ID":"e375bc72-ab6c-4c73-ac41-c31384a0623c","Type":"ContainerDied","Data":"ddcc1a618b9e7f5a3dd7bd5ab61447d4b254064a7f9cd9d385851b57f4319228"}
Oct 30 12:44:20.307216 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:20.306950    4522 scope.go:122] "RemoveContainer" containerID="33ab81c3be7ee3e2d406f0372a8347e7e15663259ef042b0147da6601888a95e"
Oct 30 12:44:20.307216 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:20.306876    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-p8qh9"
Oct 30 12:44:20.317093 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:20.317074    4522 scope.go:122] "RemoveContainer" containerID="5e0f1fcd56ad2e00ea780a93a46ea0f9b8d68a024cb3fbec28ab3c9928923425"
Oct 30 12:44:20.325412 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:20.325385    4522 scope.go:122] "RemoveContainer" containerID="33ab81c3be7ee3e2d406f0372a8347e7e15663259ef042b0147da6601888a95e"
Oct 30 12:44:20.325697 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:44:20.325674    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ab81c3be7ee3e2d406f0372a8347e7e15663259ef042b0147da6601888a95e\": container with ID starting with 33ab81c3be7ee3e2d406f0372a8347e7e15663259ef042b0147da6601888a95e not found: ID does not exist" containerID="33ab81c3be7ee3e2d406f0372a8347e7e15663259ef042b0147da6601888a95e"
Oct 30 12:44:20.325764 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:20.325704    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ab81c3be7ee3e2d406f0372a8347e7e15663259ef042b0147da6601888a95e"} err="failed to get container status \"33ab81c3be7ee3e2d406f0372a8347e7e15663259ef042b0147da6601888a95e\": rpc error: code = NotFound desc = could not find container \"33ab81c3be7ee3e2d406f0372a8347e7e15663259ef042b0147da6601888a95e\": container with ID starting with 33ab81c3be7ee3e2d406f0372a8347e7e15663259ef042b0147da6601888a95e not found: ID does not exist"
Oct 30 12:44:20.325764 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:20.325721    4522 scope.go:122] "RemoveContainer" containerID="5e0f1fcd56ad2e00ea780a93a46ea0f9b8d68a024cb3fbec28ab3c9928923425"
Oct 30 12:44:20.326019 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:44:20.326002    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0f1fcd56ad2e00ea780a93a46ea0f9b8d68a024cb3fbec28ab3c9928923425\": container with ID starting with 5e0f1fcd56ad2e00ea780a93a46ea0f9b8d68a024cb3fbec28ab3c9928923425 not found: ID does not exist" containerID="5e0f1fcd56ad2e00ea780a93a46ea0f9b8d68a024cb3fbec28ab3c9928923425"
Oct 30 12:44:20.326064 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:20.326024    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0f1fcd56ad2e00ea780a93a46ea0f9b8d68a024cb3fbec28ab3c9928923425"} err="failed to get container status \"5e0f1fcd56ad2e00ea780a93a46ea0f9b8d68a024cb3fbec28ab3c9928923425\": rpc error: code = NotFound desc = could not find container \"5e0f1fcd56ad2e00ea780a93a46ea0f9b8d68a024cb3fbec28ab3c9928923425\": container with ID starting with 5e0f1fcd56ad2e00ea780a93a46ea0f9b8d68a024cb3fbec28ab3c9928923425 not found: ID does not exist"
Oct 30 12:44:20.356123 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:20.356100    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p8qh9"]
Oct 30 12:44:20.367879 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:20.367857    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p8qh9"]
Oct 30 12:44:21.136463 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:44:21.136429    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:44:21.147749 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:21.147726    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:21.627124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:21.627085    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:22.333267 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:22.333238    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e375bc72-ab6c-4c73-ac41-c31384a0623c" path="/var/lib/kubelet/pods/e375bc72-ab6c-4c73-ac41-c31384a0623c/volumes"
Oct 30 12:44:22.923197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:22.923169    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:23.024005 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:23.023980    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:30.487196 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.487163    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"]
Oct 30 12:44:30.487585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.487470    4522 cpu_manager.go:414] "containerMap: removing container" podUID="e375bc72-ab6c-4c73-ac41-c31384a0623c" containerName="manager"
Oct 30 12:44:30.487585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.487481    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e375bc72-ab6c-4c73-ac41-c31384a0623c" containerName="manager"
Oct 30 12:44:30.487585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.487492    4522 cpu_manager.go:414] "containerMap: removing container" podUID="e375bc72-ab6c-4c73-ac41-c31384a0623c" containerName="manager"
Oct 30 12:44:30.487585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.487498    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e375bc72-ab6c-4c73-ac41-c31384a0623c" containerName="manager"
Oct 30 12:44:30.487585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.487570    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e375bc72-ab6c-4c73-ac41-c31384a0623c" containerName="manager"
Oct 30 12:44:30.487585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.487578    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e375bc72-ab6c-4c73-ac41-c31384a0623c" containerName="manager"
Oct 30 12:44:30.495411 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.495390    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.510407 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.510388    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"]
Oct 30 12:44:30.521024 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.521002    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 12:44:30.521146 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.521030    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 12:44:30.522998 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.522981    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-76488\""
Oct 30 12:44:30.522998 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.522994    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 12:44:30.536042 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.536023    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:30.569695 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.569669    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mftwr\" (UniqueName: \"kubernetes.io/projected/9f133540-b63f-449a-8377-36912a0f1b35-kube-api-access-mftwr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.569792 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.569702    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f133540-b63f-449a-8377-36912a0f1b35-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.569843 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.569822    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f133540-b63f-449a-8377-36912a0f1b35-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.569885 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.569865    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f133540-b63f-449a-8377-36912a0f1b35-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.569885 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.569880    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f133540-b63f-449a-8377-36912a0f1b35-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.569963 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.569894    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f133540-b63f-449a-8377-36912a0f1b35-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.671042 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.671016    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f133540-b63f-449a-8377-36912a0f1b35-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.671168 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.671059    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f133540-b63f-449a-8377-36912a0f1b35-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.671168 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.671083    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f133540-b63f-449a-8377-36912a0f1b35-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.671168 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.671124    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mftwr\" (UniqueName: \"kubernetes.io/projected/9f133540-b63f-449a-8377-36912a0f1b35-kube-api-access-mftwr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.671168 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.671155    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f133540-b63f-449a-8377-36912a0f1b35-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.671365 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.671213    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f133540-b63f-449a-8377-36912a0f1b35-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.671527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.671500    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f133540-b63f-449a-8377-36912a0f1b35-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.671634 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.671580    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f133540-b63f-449a-8377-36912a0f1b35-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.671634 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.671603    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f133540-b63f-449a-8377-36912a0f1b35-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.673439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.673422    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f133540-b63f-449a-8377-36912a0f1b35-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.673489 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.673456    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f133540-b63f-449a-8377-36912a0f1b35-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.689981 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.689952    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mftwr\" (UniqueName: \"kubernetes.io/projected/9f133540-b63f-449a-8377-36912a0f1b35-kube-api-access-mftwr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj\" (UID: \"9f133540-b63f-449a-8377-36912a0f1b35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:30.807786 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:30.807764    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:31.107557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:31.107529    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"]
Oct 30 12:44:31.108325 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:44:31.108305    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f133540_b63f_449a_8377_36912a0f1b35.slice/crio-f5140adda86c8ab3bcd98bd0f188d95679db74b84c3440f32722bb2c0ad2c25e WatchSource:0}: Error finding container f5140adda86c8ab3bcd98bd0f188d95679db74b84c3440f32722bb2c0ad2c25e: Status 404 returned error can't find the container with id f5140adda86c8ab3bcd98bd0f188d95679db74b84c3440f32722bb2c0ad2c25e
Oct 30 12:44:31.355460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:31.355393    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj" event={"ID":"9f133540-b63f-449a-8377-36912a0f1b35","Type":"ContainerStarted","Data":"f5140adda86c8ab3bcd98bd0f188d95679db74b84c3440f32722bb2c0ad2c25e"}
Oct 30 12:44:32.091434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:32.091384    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:44:32.362150 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:32.362062    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj" event={"ID":"9f133540-b63f-449a-8377-36912a0f1b35","Type":"ContainerStarted","Data":"74047aedc9dc1727651f70020a98ec88ee4341cd55a0875b53242522a899cc64"}
Oct 30 12:44:32.401842 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:32.401798    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj" podStartSLOduration=1.387160836 podStartE2EDuration="2.401784313s" totalImagesPullingTime="1.014623477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:32.400948905 +0000 UTC m=+516.216275396" watchObservedRunningTime="2026-10-30 12:44:32.401784313 +0000 UTC m=+516.217110802"
Oct 30 12:44:32.808684 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:32.808646    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:34.210012 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:34.209978    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:35.619486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:35.619451    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:44:35.932636 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:35.932546    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"]
Oct 30 12:44:35.936464 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:35.936433    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:35.959943 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:35.959916    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"]
Oct 30 12:44:35.964514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:35.964493    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 12:44:35.993597 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:35.993573    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:36.010675 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.010654    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.010789 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.010687    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.010789 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.010707    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8qb5\" (UniqueName: \"kubernetes.io/projected/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-kube-api-access-c8qb5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.010789 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.010726    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.010963 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.010897    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.011021 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.011004    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.112443 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.112405    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.112599 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.112456    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8qb5\" (UniqueName: \"kubernetes.io/projected/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-kube-api-access-c8qb5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.112599 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.112495    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.112599 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.112560    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.112757 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.112635    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.112757 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.112684    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.112982 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.112954    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.112982 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.112972    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.113116 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.113037    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.114704 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.114685    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.115019 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.114997    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.132472 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.132445    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8qb5\" (UniqueName: \"kubernetes.io/projected/29d90fc8-e584-45cb-9e65-dfaa8c7bc226-kube-api-access-c8qb5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-z7swk\" (UID: \"29d90fc8-e584-45cb-9e65-dfaa8c7bc226\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.250466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.250378    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:36.558343 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:36.558309    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"]
Oct 30 12:44:36.558692 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:44:36.558664    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d90fc8_e584_45cb_9e65_dfaa8c7bc226.slice/crio-d04fe8f1ecce34e669a709d3d33dac06953f1a79729e7f75f841066166f9d6c8 WatchSource:0}: Error finding container d04fe8f1ecce34e669a709d3d33dac06953f1a79729e7f75f841066166f9d6c8: Status 404 returned error can't find the container with id d04fe8f1ecce34e669a709d3d33dac06953f1a79729e7f75f841066166f9d6c8
Oct 30 12:44:37.009503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:37.009458    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:44:37.208990 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:37.208889    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:37.389197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:37.389160    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk" event={"ID":"29d90fc8-e584-45cb-9e65-dfaa8c7bc226","Type":"ContainerStarted","Data":"2893db21d20d767f95963b2c0f8d550ca31381e36e1d6be27e2f0af0f9891031"}
Oct 30 12:44:37.389335 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:37.389208    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk" event={"ID":"29d90fc8-e584-45cb-9e65-dfaa8c7bc226","Type":"ContainerStarted","Data":"d04fe8f1ecce34e669a709d3d33dac06953f1a79729e7f75f841066166f9d6c8"}
Oct 30 12:44:37.432476 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:37.432424    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk" podStartSLOduration=2.211895507 podStartE2EDuration="2.432410586s" totalImagesPullingTime="220.515079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:37.430911742 +0000 UTC m=+521.246238232" watchObservedRunningTime="2026-10-30 12:44:37.432410586 +0000 UTC m=+521.247737075"
Oct 30 12:44:38.251448 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.251413    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:38.452999 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.452965    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:44:38.564206 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.564167    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"]
Oct 30 12:44:38.567627 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.567604    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.586590 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.586567    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 12:44:38.600248 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.600217    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"]
Oct 30 12:44:38.635377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.635319    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt99n\" (UniqueName: \"kubernetes.io/projected/f2e6d063-6dbf-4f1b-9f07-01c668500c25-kube-api-access-rt99n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.635521 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.635452    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e6d063-6dbf-4f1b-9f07-01c668500c25-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.635521 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.635493    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e6d063-6dbf-4f1b-9f07-01c668500c25-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.635521 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.635511    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e6d063-6dbf-4f1b-9f07-01c668500c25-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.635645 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.635545    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e6d063-6dbf-4f1b-9f07-01c668500c25-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.635700 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.635677    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e6d063-6dbf-4f1b-9f07-01c668500c25-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.645772 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.645738    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:38.737108 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.737074    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e6d063-6dbf-4f1b-9f07-01c668500c25-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.737279 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.737114    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e6d063-6dbf-4f1b-9f07-01c668500c25-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.737279 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.737158    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt99n\" (UniqueName: \"kubernetes.io/projected/f2e6d063-6dbf-4f1b-9f07-01c668500c25-kube-api-access-rt99n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.737279 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.737209    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e6d063-6dbf-4f1b-9f07-01c668500c25-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.737444 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.737342    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e6d063-6dbf-4f1b-9f07-01c668500c25-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.737444 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.737383    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e6d063-6dbf-4f1b-9f07-01c668500c25-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.737563 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.737544    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e6d063-6dbf-4f1b-9f07-01c668500c25-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.737632 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.737609    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e6d063-6dbf-4f1b-9f07-01c668500c25-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.737766 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.737745    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e6d063-6dbf-4f1b-9f07-01c668500c25-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.739546 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.739522    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e6d063-6dbf-4f1b-9f07-01c668500c25-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.739796 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.739780    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e6d063-6dbf-4f1b-9f07-01c668500c25-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.760604 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.760580    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt99n\" (UniqueName: \"kubernetes.io/projected/f2e6d063-6dbf-4f1b-9f07-01c668500c25-kube-api-access-rt99n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz\" (UID: \"f2e6d063-6dbf-4f1b-9f07-01c668500c25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:38.883598 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:38.883509    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:39.195829 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:39.195801    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"]
Oct 30 12:44:39.197794 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:44:39.197766    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e6d063_6dbf_4f1b_9f07_01c668500c25.slice/crio-66f1379d4913cc7c9c3f3f7b29959b64f65fab0be196d7965ce14b78c582578e WatchSource:0}: Error finding container 66f1379d4913cc7c9c3f3f7b29959b64f65fab0be196d7965ce14b78c582578e: Status 404 returned error can't find the container with id 66f1379d4913cc7c9c3f3f7b29959b64f65fab0be196d7965ce14b78c582578e
Oct 30 12:44:39.398595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:39.398558    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz" event={"ID":"f2e6d063-6dbf-4f1b-9f07-01c668500c25","Type":"ContainerStarted","Data":"66f1379d4913cc7c9c3f3f7b29959b64f65fab0be196d7965ce14b78c582578e"}
Oct 30 12:44:39.848279 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:39.848229    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:40.403787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:40.403748    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz" event={"ID":"f2e6d063-6dbf-4f1b-9f07-01c668500c25","Type":"ContainerStarted","Data":"c432428f215a66f5cd70d7d4936bc218d42b27d7acde262cd9d3f15420280085"}
Oct 30 12:44:40.442842 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:40.442813    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:44:40.474935 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:40.474877    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz" podStartSLOduration=2.264938476 podStartE2EDuration="2.47486291s" totalImagesPullingTime="209.924434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:40.473942835 +0000 UTC m=+524.289269323" watchObservedRunningTime="2026-10-30 12:44:40.47486291 +0000 UTC m=+524.290189392"
Oct 30 12:44:40.808610 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:40.808572    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:40.822538 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:40.822509    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:40.885063 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:40.885032    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:41.418465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:41.418436    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj"
Oct 30 12:44:42.253232 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:42.253193    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:44:42.915737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:42.915700    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:44.401971 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.401941    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"]
Oct 30 12:44:44.405244 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.405229    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.432944 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.432916    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 12:44:44.434321 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.434298    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"]
Oct 30 12:44:44.493086 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.493055    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:44.512249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.512219    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce95115a-fa86-4eab-9f15-1f455288c014-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.512387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.512266    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce95115a-fa86-4eab-9f15-1f455288c014-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.512454 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.512401    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmm6h\" (UniqueName: \"kubernetes.io/projected/ce95115a-fa86-4eab-9f15-1f455288c014-kube-api-access-qmm6h\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.512454 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.512439    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce95115a-fa86-4eab-9f15-1f455288c014-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.512558 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.512497    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce95115a-fa86-4eab-9f15-1f455288c014-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.512558 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.512539    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce95115a-fa86-4eab-9f15-1f455288c014-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.613210 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.613181    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce95115a-fa86-4eab-9f15-1f455288c014-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.613394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.613236    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce95115a-fa86-4eab-9f15-1f455288c014-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.613394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.613253    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce95115a-fa86-4eab-9f15-1f455288c014-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.613394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.613281    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce95115a-fa86-4eab-9f15-1f455288c014-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.613394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.613317    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce95115a-fa86-4eab-9f15-1f455288c014-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.613394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.613368    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmm6h\" (UniqueName: \"kubernetes.io/projected/ce95115a-fa86-4eab-9f15-1f455288c014-kube-api-access-qmm6h\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.613772 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.613681    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce95115a-fa86-4eab-9f15-1f455288c014-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.613772 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.613691    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce95115a-fa86-4eab-9f15-1f455288c014-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.613869 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.613844    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce95115a-fa86-4eab-9f15-1f455288c014-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.615713 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.615694    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce95115a-fa86-4eab-9f15-1f455288c014-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.615791 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.615697    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce95115a-fa86-4eab-9f15-1f455288c014-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.633346 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.633321    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmm6h\" (UniqueName: \"kubernetes.io/projected/ce95115a-fa86-4eab-9f15-1f455288c014-kube-api-access-qmm6h\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq\" (UID: \"ce95115a-fa86-4eab-9f15-1f455288c014\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:44.666736 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.666679    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:44:44.717518 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:44.717484    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:45.017828 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:44:45.017794    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce95115a_fa86_4eab_9f15_1f455288c014.slice/crio-cea1a917e80553f1754e2345b840c1a1743c9b088d58555f02651e636e3fef80 WatchSource:0}: Error finding container cea1a917e80553f1754e2345b840c1a1743c9b088d58555f02651e636e3fef80: Status 404 returned error can't find the container with id cea1a917e80553f1754e2345b840c1a1743c9b088d58555f02651e636e3fef80
Oct 30 12:44:45.017946 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:45.017912    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"]
Oct 30 12:44:45.429437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:45.429399    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" event={"ID":"ce95115a-fa86-4eab-9f15-1f455288c014","Type":"ContainerStarted","Data":"bc2c7424ce97419d70eb11f60ed4cf95b13b34621fb446da623d4bc2782916ac"}
Oct 30 12:44:45.429849 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:45.429449    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" event={"ID":"ce95115a-fa86-4eab-9f15-1f455288c014","Type":"ContainerStarted","Data":"cea1a917e80553f1754e2345b840c1a1743c9b088d58555f02651e636e3fef80"}
Oct 30 12:44:45.467434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:45.467386    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podStartSLOduration=1.27194135 podStartE2EDuration="1.467347342s" totalImagesPullingTime="195.405992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:45.467008483 +0000 UTC m=+529.282334986" watchObservedRunningTime="2026-10-30 12:44:45.467347342 +0000 UTC m=+529.282673831"
Oct 30 12:44:45.718274 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:45.718183    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:46.251286 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:46.251250    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:46.263561 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:46.263533    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:46.444411 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:46.444385    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk"
Oct 30 12:44:46.834007 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:46.833978    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:47.038538 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:47.038501    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:44:48.884581 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:48.884544    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:48.896826 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:48.896798    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:49.457534 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:49.457511    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz"
Oct 30 12:44:51.134545 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:44:51.134503    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:44:54.717829 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:54.717787    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:54.730392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:54.730341    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:55.481036 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:55.481009    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:44:58.841552 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:58.841511    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"]
Oct 30 12:44:58.846585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:58.846555    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:58.862825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:58.862800    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bn28m\""
Oct 30 12:44:58.875633 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:58.875605    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"]
Oct 30 12:44:58.879591 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:58.879567    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"]
Oct 30 12:44:58.879698 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:58.879673    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:58.901985 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:58.901948    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"]
Oct 30 12:44:59.030864 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.030837    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.031040 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.030876    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.031040 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.030892    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90570573-85f7-4227-b6fe-86abaf0804e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.031040 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.030909    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.031040 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031006    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psjs7\" (UniqueName: \"kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-kube-api-access-psjs7\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.031186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031114    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90570573-85f7-4227-b6fe-86abaf0804e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.031186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031163    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.031186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031183    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.031312 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031206    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.031312 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031232    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.031440 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031333    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.031440 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031397    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.031440 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031431    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swv8x\" (UniqueName: \"kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-kube-api-access-swv8x\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.031570 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031461    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.031617 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031566    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.031617 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031603    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.031683 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031640    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.031721 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.031677    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.132099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132043    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.132099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132071    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.132099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132091    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.132310 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132106    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.132310 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132124    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.132310 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132139    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.132310 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132158    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swv8x\" (UniqueName: \"kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-kube-api-access-swv8x\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.132690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132317    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.132690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132387    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.132690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132413    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.132690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132434    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.132690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132507    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.132690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132547    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.132690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132560    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.132690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132549    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.132690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132608    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.132690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132626    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.132690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132624    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.132690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132687    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.133240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132712    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90570573-85f7-4227-b6fe-86abaf0804e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.133240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132736    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.133240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132741    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.133240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132762    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psjs7\" (UniqueName: \"kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-kube-api-access-psjs7\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.133240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132793    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90570573-85f7-4227-b6fe-86abaf0804e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.133240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132928    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.133240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.132961    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.133240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.133223    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.133708 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.133410    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90570573-85f7-4227-b6fe-86abaf0804e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.134852 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.134825    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.134984 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.134876    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.135193 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.135175    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90570573-85f7-4227-b6fe-86abaf0804e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.135253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.135201    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.152114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.152083    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.152241 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.152200    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swv8x\" (UniqueName: \"kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-kube-api-access-swv8x\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nllbj\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.152619 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.152596    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.152663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.152628    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psjs7\" (UniqueName: \"kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-kube-api-access-psjs7\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8d5wt\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.159464 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.159448    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:44:59.195131 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.195101    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:44:59.480857 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.480799    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"]
Oct 30 12:44:59.487106 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:44:59.487078    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4a185e_9025_45b1_bb95_c8e86a53e9bc.slice/crio-79b2b72e37c7a3c9d212263c70a5291c8d7876ad97a6c51a4b9d1114b0f04396 WatchSource:0}: Error finding container 79b2b72e37c7a3c9d212263c70a5291c8d7876ad97a6c51a4b9d1114b0f04396: Status 404 returned error can't find the container with id 79b2b72e37c7a3c9d212263c70a5291c8d7876ad97a6c51a4b9d1114b0f04396
Oct 30 12:44:59.501129 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:44:59.501106    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"]
Oct 30 12:44:59.501790 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:44:59.501769    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90570573_85f7_4227_b6fe_86abaf0804e5.slice/crio-669aef18a2416057cb82c7bf7e615b10b963077f956de554b3cb44de8f3feb90 WatchSource:0}: Error finding container 669aef18a2416057cb82c7bf7e615b10b963077f956de554b3cb44de8f3feb90: Status 404 returned error can't find the container with id 669aef18a2416057cb82c7bf7e615b10b963077f956de554b3cb44de8f3feb90
Oct 30 12:45:00.153999 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.153970    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"]
Oct 30 12:45:00.157608 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.157585    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"
Oct 30 12:45:00.172633 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.172593    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:45:00.172761 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.172670    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:45:00.172838 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.172780    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 12:45:00.174611 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.174589    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"]
Oct 30 12:45:00.343061 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.343025    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889405-d7gn6\" (UID: \"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"
Oct 30 12:45:00.343248 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.343125    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78bv6\" (UniqueName: \"kubernetes.io/projected/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-kube-api-access-78bv6\") pod \"maas-api-key-cleanup-29889405-d7gn6\" (UID: \"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"
Oct 30 12:45:00.444408 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.444319    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bv6\" (UniqueName: \"kubernetes.io/projected/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-kube-api-access-78bv6\") pod \"maas-api-key-cleanup-29889405-d7gn6\" (UID: \"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"
Oct 30 12:45:00.444408 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.444404    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889405-d7gn6\" (UID: \"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"
Oct 30 12:45:00.445136 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.445090    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889405-d7gn6\" (UID: \"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"
Oct 30 12:45:00.480067 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.480039    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78bv6\" (UniqueName: \"kubernetes.io/projected/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-kube-api-access-78bv6\") pod \"maas-api-key-cleanup-29889405-d7gn6\" (UID: \"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"
Oct 30 12:45:00.491627 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.491577    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj" event={"ID":"90570573-85f7-4227-b6fe-86abaf0804e5","Type":"ContainerStarted","Data":"669aef18a2416057cb82c7bf7e615b10b963077f956de554b3cb44de8f3feb90"}
Oct 30 12:45:00.492909 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.492874    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt" event={"ID":"ba4a185e-9025-45b1-bb95-c8e86a53e9bc","Type":"ContainerStarted","Data":"79b2b72e37c7a3c9d212263c70a5291c8d7876ad97a6c51a4b9d1114b0f04396"}
Oct 30 12:45:00.773275 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:00.773251    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"
Oct 30 12:45:01.072095 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:01.072066    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"]
Oct 30 12:45:01.073954 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:45:01.073921    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683cf6fc_94c0_4d89_a18b_7b145e4bc5b5.slice/crio-e692291b737bfecebc81da4e8cffe69287e5fc59f7d405a2e74faf9aebd181a9 WatchSource:0}: Error finding container e692291b737bfecebc81da4e8cffe69287e5fc59f7d405a2e74faf9aebd181a9: Status 404 returned error can't find the container with id e692291b737bfecebc81da4e8cffe69287e5fc59f7d405a2e74faf9aebd181a9
Oct 30 12:45:01.137936 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:01.137914    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:01.498903 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:01.498831    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6" event={"ID":"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5","Type":"ContainerStarted","Data":"e692291b737bfecebc81da4e8cffe69287e5fc59f7d405a2e74faf9aebd181a9"}
Oct 30 12:45:01.844587 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:01.844548    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:02.002793 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:02.002758    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:45:02.002906 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:02.002834    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:45:02.002906 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:02.002877    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:45:02.504374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:02.504322    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj" event={"ID":"90570573-85f7-4227-b6fe-86abaf0804e5","Type":"ContainerStarted","Data":"48839364e34e432fbe676dc951db044ad8da72a6571b5dd806de83ff3215942b"}
Oct 30 12:45:02.561224 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:02.561157    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj" podStartSLOduration=2.062615055 podStartE2EDuration="4.561136989s" totalImagesPullingTime="2.498521934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.559988437 +0000 UTC m=+546.375314927" watchObservedRunningTime="2026-10-30 12:45:02.561136989 +0000 UTC m=+546.376463478"
Oct 30 12:45:03.195864 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:03.195828    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:45:03.197288 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:03.197256    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nllbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 12:45:03.197468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:03.197318    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj" podUID="90570573-85f7-4227-b6fe-86abaf0804e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 12:45:03.964565 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:03.952984    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8fd5b54-cwv87"]
Oct 30 12:45:03.964565 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:03.964047    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-8fd5b54-cwv87" podUID="d7d9458e-e3cf-47df-a993-cdc9a4e8d768" containerName="authorino" containerID="cri-o://ae97b64af26b7c2a68e1004b543d126b6a64c203cee942122440db6407b4c6ae" gracePeriod=30
Oct 30 12:45:04.196095 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.196062    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nllbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 12:45:04.196263 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.196128    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj" podUID="90570573-85f7-4227-b6fe-86abaf0804e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 12:45:04.243411 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.243373    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8fd5b54-cwv87"
Oct 30 12:45:04.401446 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.400854    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-kube-api-access-v8blz\" (UniqueName: \"kubernetes.io/projected/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-kube-api-access-v8blz\") pod \"d7d9458e-e3cf-47df-a993-cdc9a4e8d768\" (UID: \"d7d9458e-e3cf-47df-a993-cdc9a4e8d768\") "
Oct 30 12:45:04.401446 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.400906    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-tls-cert\" (UniqueName: \"kubernetes.io/secret/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-tls-cert\") pod \"d7d9458e-e3cf-47df-a993-cdc9a4e8d768\" (UID: \"d7d9458e-e3cf-47df-a993-cdc9a4e8d768\") "
Oct 30 12:45:04.405438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.405396    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-kube-api-access-v8blz" pod "d7d9458e-e3cf-47df-a993-cdc9a4e8d768" (UID: "d7d9458e-e3cf-47df-a993-cdc9a4e8d768"). InnerVolumeSpecName "kube-api-access-v8blz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:04.414246 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.414220    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-tls-cert" pod "d7d9458e-e3cf-47df-a993-cdc9a4e8d768" (UID: "d7d9458e-e3cf-47df-a993-cdc9a4e8d768"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:45:04.502201 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.502141    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8blz\" (UniqueName: \"kubernetes.io/projected/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-kube-api-access-v8blz\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:45:04.502201 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.502168    4522 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7d9458e-e3cf-47df-a993-cdc9a4e8d768-tls-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:45:04.514338 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.514306    4522 generic.go:356] "Generic (PLEG): container finished" podID="d7d9458e-e3cf-47df-a993-cdc9a4e8d768" containerID="ae97b64af26b7c2a68e1004b543d126b6a64c203cee942122440db6407b4c6ae" exitCode=0
Oct 30 12:45:04.514475 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.514365    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8fd5b54-cwv87"
Oct 30 12:45:04.514475 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.514399    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8fd5b54-cwv87" event={"ID":"d7d9458e-e3cf-47df-a993-cdc9a4e8d768","Type":"ContainerDied","Data":"ae97b64af26b7c2a68e1004b543d126b6a64c203cee942122440db6407b4c6ae"}
Oct 30 12:45:04.514475 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.514449    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8fd5b54-cwv87" event={"ID":"d7d9458e-e3cf-47df-a993-cdc9a4e8d768","Type":"ContainerDied","Data":"0b1791569404749afd53c2a5cd5b72ccf577302e08bbace7ac06905c624a4e50"}
Oct 30 12:45:04.514635 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.514477    4522 scope.go:122] "RemoveContainer" containerID="ae97b64af26b7c2a68e1004b543d126b6a64c203cee942122440db6407b4c6ae"
Oct 30 12:45:04.522686 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.522670    4522 scope.go:122] "RemoveContainer" containerID="ae97b64af26b7c2a68e1004b543d126b6a64c203cee942122440db6407b4c6ae"
Oct 30 12:45:04.522971 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:45:04.522934    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae97b64af26b7c2a68e1004b543d126b6a64c203cee942122440db6407b4c6ae\": container with ID starting with ae97b64af26b7c2a68e1004b543d126b6a64c203cee942122440db6407b4c6ae not found: ID does not exist" containerID="ae97b64af26b7c2a68e1004b543d126b6a64c203cee942122440db6407b4c6ae"
Oct 30 12:45:04.523019 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.522979    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae97b64af26b7c2a68e1004b543d126b6a64c203cee942122440db6407b4c6ae"} err="failed to get container status \"ae97b64af26b7c2a68e1004b543d126b6a64c203cee942122440db6407b4c6ae\": rpc error: code = NotFound desc = could not find container \"ae97b64af26b7c2a68e1004b543d126b6a64c203cee942122440db6407b4c6ae\": container with ID starting with ae97b64af26b7c2a68e1004b543d126b6a64c203cee942122440db6407b4c6ae not found: ID does not exist"
Oct 30 12:45:04.590825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.590797    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8fd5b54-cwv87"]
Oct 30 12:45:04.601847 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:04.601827    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8fd5b54-cwv87"]
Oct 30 12:45:05.195659 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:05.195620    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nllbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 12:45:05.196027 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:05.195667    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj" podUID="90570573-85f7-4227-b6fe-86abaf0804e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 12:45:06.195675 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:06.195632    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nllbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 12:45:06.196120 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:06.195694    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj" podUID="90570573-85f7-4227-b6fe-86abaf0804e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 12:45:06.332795 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:06.332763    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d9458e-e3cf-47df-a993-cdc9a4e8d768" path="/var/lib/kubelet/pods/d7d9458e-e3cf-47df-a993-cdc9a4e8d768/volumes"
Oct 30 12:45:07.195445 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:07.195406    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nllbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 12:45:07.195627 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:07.195462    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj" podUID="90570573-85f7-4227-b6fe-86abaf0804e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 12:45:07.636080 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:07.636042    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:08.196462 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:08.196425    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nllbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 12:45:08.196618 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:08.196489    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj" podUID="90570573-85f7-4227-b6fe-86abaf0804e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 12:45:09.195969 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:09.195929    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nllbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 12:45:09.196499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:09.195993    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj" podUID="90570573-85f7-4227-b6fe-86abaf0804e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 12:45:10.244928 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:10.244881    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:45:10.245426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:10.245296    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:45:10.246531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:10.246490    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:45:12.198758 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:12.198718    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:45:12.199072 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:12.198786    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:45:12.199072 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:12.198815    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:45:12.552581 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:12.552543    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt" event={"ID":"ba4a185e-9025-45b1-bb95-c8e86a53e9bc","Type":"ContainerStarted","Data":"6488c9cbad52d5580d8d0a0a731c94f50867c340224bbb6d8e6ec5c1b1a224ad"}
Oct 30 12:45:12.617169 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:12.617130    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt" podStartSLOduration=1.907884613 podStartE2EDuration="14.617118702s" totalImagesPullingTime="12.709234089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:12.605079798 +0000 UTC m=+556.420406285" watchObservedRunningTime="2026-10-30 12:45:12.617118702 +0000 UTC m=+556.432445189"
Oct 30 12:45:13.160536 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.160498    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:45:13.161974 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.161949    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8d5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:45:13.162096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.162000    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt" podUID="ba4a185e-9025-45b1-bb95-c8e86a53e9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 12:45:13.907773 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.907740    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"]
Oct 30 12:45:13.908304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.908283    4522 cpu_manager.go:414] "containerMap: removing container" podUID="d7d9458e-e3cf-47df-a993-cdc9a4e8d768" containerName="authorino"
Oct 30 12:45:13.908396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.908308    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d9458e-e3cf-47df-a993-cdc9a4e8d768" containerName="authorino"
Oct 30 12:45:13.908447 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.908424    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d9458e-e3cf-47df-a993-cdc9a4e8d768" containerName="authorino"
Oct 30 12:45:13.911691 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.911669    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:13.934969 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.934923    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"]
Oct 30 12:45:13.995876 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.995847    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:13.996049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.995909    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:13.996049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.995947    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:13.996049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.996008    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:13.996049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.996030    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f352699b-bc5c-4be3-9f32-ebdec5169271-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:13.996275 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.996055    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f352699b-bc5c-4be3-9f32-ebdec5169271-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:13.996275 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.996088    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2bj2\" (UniqueName: \"kubernetes.io/projected/f352699b-bc5c-4be3-9f32-ebdec5169271-kube-api-access-j2bj2\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:13.996275 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.996131    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:13.996275 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:13.996165    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f352699b-bc5c-4be3-9f32-ebdec5169271-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.096625 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.096585    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.096787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.096646    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.096787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.096669    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.096787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.096717    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.096787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.096741    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f352699b-bc5c-4be3-9f32-ebdec5169271-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.096787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.096769    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f352699b-bc5c-4be3-9f32-ebdec5169271-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.097046 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.096799    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2bj2\" (UniqueName: \"kubernetes.io/projected/f352699b-bc5c-4be3-9f32-ebdec5169271-kube-api-access-j2bj2\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.097046 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.096844    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.097046 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.096876    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f352699b-bc5c-4be3-9f32-ebdec5169271-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.097046 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.096981    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.097608 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.097578    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.097813 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.097789    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f352699b-bc5c-4be3-9f32-ebdec5169271-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.098026 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.098007    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.098113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.098089    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.099458 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.099432    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f352699b-bc5c-4be3-9f32-ebdec5169271-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.100785 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.100747    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f352699b-bc5c-4be3-9f32-ebdec5169271-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.115702 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.115679    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f352699b-bc5c-4be3-9f32-ebdec5169271-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.115796 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.115772    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2bj2\" (UniqueName: \"kubernetes.io/projected/f352699b-bc5c-4be3-9f32-ebdec5169271-kube-api-access-j2bj2\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9twbb\" (UID: \"f352699b-bc5c-4be3-9f32-ebdec5169271\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.160541 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.160485    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:45:14.160787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.160763    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8d5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:45:14.160899 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.160815    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt" podUID="ba4a185e-9025-45b1-bb95-c8e86a53e9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 12:45:14.229167 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.229137    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:14.593378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.591066    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"]
Oct 30 12:45:14.597626 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:45:14.597584    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf352699b_bc5c_4be3_9f32_ebdec5169271.slice/crio-edb5744d3fb349933d229703747941636ec55e0ada2ad8416f6b62d32812c505 WatchSource:0}: Error finding container edb5744d3fb349933d229703747941636ec55e0ada2ad8416f6b62d32812c505: Status 404 returned error can't find the container with id edb5744d3fb349933d229703747941636ec55e0ada2ad8416f6b62d32812c505
Oct 30 12:45:14.600377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.600323    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:45:14.600479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.600422    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:45:14.600479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.600468    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:45:14.993423 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:14.993283    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:15.098381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:15.097878    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:15.160233 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:15.160198    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8d5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:45:15.160415 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:15.160267    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt" podUID="ba4a185e-9025-45b1-bb95-c8e86a53e9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 12:45:15.592513 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:15.592476    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb" event={"ID":"f352699b-bc5c-4be3-9f32-ebdec5169271","Type":"ContainerStarted","Data":"c665ce299ab99dc70edc62367f0245cdbba37026a48bbdb55dadb984b3fb0902"}
Oct 30 12:45:15.592812 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:15.592790    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb" event={"ID":"f352699b-bc5c-4be3-9f32-ebdec5169271","Type":"ContainerStarted","Data":"edb5744d3fb349933d229703747941636ec55e0ada2ad8416f6b62d32812c505"}
Oct 30 12:45:16.159837 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:16.159803    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8d5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:45:16.160253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:16.159857    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt" podUID="ba4a185e-9025-45b1-bb95-c8e86a53e9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 12:45:16.229635 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:16.229597    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:16.231113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:16.231090    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9twbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:45:16.231237 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:16.231130    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb" podUID="f352699b-bc5c-4be3-9f32-ebdec5169271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:45:16.621546 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:16.621511    4522 generic.go:356] "Generic (PLEG): container finished" podID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" containerID="2a5e5fa1302c076880abc7bb11fd4ecd79e75212288873006ca6ea10e0b129c8" exitCode=60
Oct 30 12:45:16.622238 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:16.622215    4522 scope.go:122] "RemoveContainer" containerID="2a5e5fa1302c076880abc7bb11fd4ecd79e75212288873006ca6ea10e0b129c8"
Oct 30 12:45:16.622483 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:16.622462    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6" event={"ID":"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5","Type":"ContainerDied","Data":"2a5e5fa1302c076880abc7bb11fd4ecd79e75212288873006ca6ea10e0b129c8"}
Oct 30 12:45:16.659656 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:16.659618    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb" podStartSLOduration=3.659606805 podStartE2EDuration="3.659606805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:15.660517095 +0000 UTC m=+559.475843582" watchObservedRunningTime="2026-10-30 12:45:16.659606805 +0000 UTC m=+560.474933294"
Oct 30 12:45:17.160586 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:17.160505    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8d5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:45:17.160586 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:17.160555    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt" podUID="ba4a185e-9025-45b1-bb95-c8e86a53e9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 12:45:17.229982 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:17.229944    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9twbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:45:17.230144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:17.229993    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb" podUID="f352699b-bc5c-4be3-9f32-ebdec5169271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:45:17.628327 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:17.628295    4522 generic.go:356] "Generic (PLEG): container finished" podID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" containerID="40cb3c518badcc8c60ecc18563be28ac5d21e4f6df3074286761e6d39e060bad" exitCode=60
Oct 30 12:45:17.628487 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:17.628401    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6" event={"ID":"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5","Type":"ContainerDied","Data":"40cb3c518badcc8c60ecc18563be28ac5d21e4f6df3074286761e6d39e060bad"}
Oct 30 12:45:17.628487 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:17.628465    4522 scope.go:122] "RemoveContainer" containerID="2a5e5fa1302c076880abc7bb11fd4ecd79e75212288873006ca6ea10e0b129c8"
Oct 30 12:45:17.628893 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:17.628875    4522 scope.go:122] "RemoveContainer" containerID="40cb3c518badcc8c60ecc18563be28ac5d21e4f6df3074286761e6d39e060bad"
Oct 30 12:45:17.629202 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:45:17.629180    4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889405-d7gn6_odh-ai-gateway-infra(683cf6fc-94c0-4d89-a18b-7b145e4bc5b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6" podUID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5"
Oct 30 12:45:18.160528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:18.160486    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8d5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:45:18.160709 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:18.160551    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt" podUID="ba4a185e-9025-45b1-bb95-c8e86a53e9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 12:45:18.229993 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:18.229955    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9twbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:45:18.230156 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:18.230011    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb" podUID="f352699b-bc5c-4be3-9f32-ebdec5169271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:45:18.635074 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:18.635044    4522 scope.go:122] "RemoveContainer" containerID="40cb3c518badcc8c60ecc18563be28ac5d21e4f6df3074286761e6d39e060bad"
Oct 30 12:45:18.635375 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:45:18.635329    4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889405-d7gn6_odh-ai-gateway-infra(683cf6fc-94c0-4d89-a18b-7b145e4bc5b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6" podUID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5"
Oct 30 12:45:19.160454 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:19.160412    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8d5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 12:45:19.160604 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:19.160483    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt" podUID="ba4a185e-9025-45b1-bb95-c8e86a53e9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 12:45:19.230129 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:19.230095    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9twbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:45:19.230508 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:19.230148    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb" podUID="f352699b-bc5c-4be3-9f32-ebdec5169271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:45:20.229732 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:20.229698    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9twbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:45:20.229936 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:20.229762    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb" podUID="f352699b-bc5c-4be3-9f32-ebdec5169271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:45:20.833121 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:20.833084    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:45:20.834442 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:20.834413    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:45:21.133319 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:45:21.133202    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:45:21.229935 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:21.229904    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9twbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:45:21.230058 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:21.229959    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb" podUID="f352699b-bc5c-4be3-9f32-ebdec5169271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:45:22.230296 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:22.230259    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9twbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:45:22.230741 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:22.230321    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb" podUID="f352699b-bc5c-4be3-9f32-ebdec5169271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:45:23.234058 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:23.234029    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:23.234528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:23.234270    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:23.235012 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:23.234990    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9twbb"
Oct 30 12:45:26.495977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:26.495940    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:26.607981 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:26.607955    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:28.327742 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:28.327701    4522 scope.go:122] "RemoveContainer" containerID="40cb3c518badcc8c60ecc18563be28ac5d21e4f6df3074286761e6d39e060bad"
Oct 30 12:45:28.679374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:28.679305    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6" event={"ID":"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5","Type":"ContainerStarted","Data":"1cf9f8cb45e0296d76996875de5bff6e52c4740926ccfaf5234d7a4de3bce855"}
Oct 30 12:45:28.720733 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:28.720687    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6" podStartSLOduration=14.369495608 podStartE2EDuration="28.720676254s" totalImagesPullingTime="14.351180646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:28.720224132 +0000 UTC m=+572.535550619" watchObservedRunningTime="2026-10-30 12:45:28.720676254 +0000 UTC m=+572.536002760"
Oct 30 12:45:29.004820 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.004797    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"]
Oct 30 12:45:29.008177 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.008147    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.036019 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.035990    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"]
Oct 30 12:45:29.046087 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.046060    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.046200 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.046131    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.046200 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.046177    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae991ed0-e51c-4804-b9c3-31e0152a6170-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.046292 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.046272    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khb8b\" (UniqueName: \"kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-kube-api-access-khb8b\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.046342 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.046304    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.046419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.046387    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.046474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.046422    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.046474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.046447    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.046543 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.046489    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.147189 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.147152    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.147381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.147196    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.147381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.147247    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.147381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.147343    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.147571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.147413    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.147571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.147434    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae991ed0-e51c-4804-b9c3-31e0152a6170-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.147571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.147479    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khb8b\" (UniqueName: \"kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-kube-api-access-khb8b\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.147571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.147510    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.147571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.147532    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.147826 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.147807    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.148286 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.147985    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.148286 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.148119    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.148286 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.148189    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.148527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.148345    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae991ed0-e51c-4804-b9c3-31e0152a6170-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.149986 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.149959    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.150202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.150179    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.167265 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.167239    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.167829 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.167811    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khb8b\" (UniqueName: \"kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-kube-api-access-khb8b\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j2w5h\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.346210 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.346185    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:29.653929 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.653903    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"]
Oct 30 12:45:29.654666 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:45:29.654645    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae991ed0_e51c_4804_b9c3_31e0152a6170.slice/crio-27beb08316d108514dcde220c16cc4534eeed13df54ce1270f272b35922acc0f WatchSource:0}: Error finding container 27beb08316d108514dcde220c16cc4534eeed13df54ce1270f272b35922acc0f: Status 404 returned error can't find the container with id 27beb08316d108514dcde220c16cc4534eeed13df54ce1270f272b35922acc0f
Oct 30 12:45:29.656905 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.656869    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:45:29.657000 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.656932    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:45:29.657000 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.656959    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:45:29.685373 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.685326    4522 generic.go:356] "Generic (PLEG): container finished" podID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" containerID="1cf9f8cb45e0296d76996875de5bff6e52c4740926ccfaf5234d7a4de3bce855" exitCode=60
Oct 30 12:45:29.685466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.685408    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6" event={"ID":"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5","Type":"ContainerDied","Data":"1cf9f8cb45e0296d76996875de5bff6e52c4740926ccfaf5234d7a4de3bce855"}
Oct 30 12:45:29.685525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.685467    4522 scope.go:122] "RemoveContainer" containerID="40cb3c518badcc8c60ecc18563be28ac5d21e4f6df3074286761e6d39e060bad"
Oct 30 12:45:29.685760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.685743    4522 scope.go:122] "RemoveContainer" containerID="1cf9f8cb45e0296d76996875de5bff6e52c4740926ccfaf5234d7a4de3bce855"
Oct 30 12:45:29.685960 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:45:29.685938    4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889405-d7gn6_odh-ai-gateway-infra(683cf6fc-94c0-4d89-a18b-7b145e4bc5b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6" podUID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5"
Oct 30 12:45:29.686764 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.686745    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h" event={"ID":"ae991ed0-e51c-4804-b9c3-31e0152a6170","Type":"ContainerStarted","Data":"27beb08316d108514dcde220c16cc4534eeed13df54ce1270f272b35922acc0f"}
Oct 30 12:45:29.732010 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:29.731984    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"]
Oct 30 12:45:30.697934 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:30.697896    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h" event={"ID":"ae991ed0-e51c-4804-b9c3-31e0152a6170","Type":"ContainerStarted","Data":"2c3c5ba36735b6e8d3dbfc93a092da964414098fd28a3dc8c5eeb6ab6759d490"}
Oct 30 12:45:30.795382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:30.793145    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h" podStartSLOduration=2.793128167 podStartE2EDuration="2.793128167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:30.776775933 +0000 UTC m=+574.592102434" watchObservedRunningTime="2026-10-30 12:45:30.793128167 +0000 UTC m=+574.608454655"
Oct 30 12:45:30.844595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:30.844571    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"
Oct 30 12:45:30.966022 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:30.965946    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-maas-api-service-ca\") pod \"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5\" (UID: \"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5\") "
Oct 30 12:45:30.966161 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:30.966033    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-kube-api-access-78bv6\" (UniqueName: \"kubernetes.io/projected/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-kube-api-access-78bv6\") pod \"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5\" (UID: \"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5\") "
Oct 30 12:45:30.966268 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:30.966248    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-maas-api-service-ca" pod "683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" (UID: "683cf6fc-94c0-4d89-a18b-7b145e4bc5b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:30.992964 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:30.992932    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-kube-api-access-78bv6" pod "683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" (UID: "683cf6fc-94c0-4d89-a18b-7b145e4bc5b5"). InnerVolumeSpecName "kube-api-access-78bv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:31.088373 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:31.068071    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78bv6\" (UniqueName: \"kubernetes.io/projected/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-kube-api-access-78bv6\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:45:31.088373 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:31.068313    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5-maas-api-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:45:31.347279 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:31.347247    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:31.348688 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:31.348665    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-j2w5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 12:45:31.348799 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:31.348710    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h" podUID="ae991ed0-e51c-4804-b9c3-31e0152a6170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 12:45:31.703578 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:31.703502    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"
Oct 30 12:45:31.703578 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:31.703541    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6" event={"ID":"683cf6fc-94c0-4d89-a18b-7b145e4bc5b5","Type":"ContainerDied","Data":"e692291b737bfecebc81da4e8cffe69287e5fc59f7d405a2e74faf9aebd181a9"}
Oct 30 12:45:31.704000 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:31.703599    4522 scope.go:122] "RemoveContainer" containerID="1cf9f8cb45e0296d76996875de5bff6e52c4740926ccfaf5234d7a4de3bce855"
Oct 30 12:45:31.752261 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:31.752234    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"]
Oct 30 12:45:31.763918 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:31.763895    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-d7gn6"]
Oct 30 12:45:32.332557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:32.332528    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" path="/var/lib/kubelet/pods/683cf6fc-94c0-4d89-a18b-7b145e4bc5b5/volumes"
Oct 30 12:45:32.347251 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:32.347217    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-j2w5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 12:45:32.347400 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:32.347262    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h" podUID="ae991ed0-e51c-4804-b9c3-31e0152a6170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 12:45:33.347505 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:33.347470    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-j2w5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 12:45:33.347895 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:33.347517    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h" podUID="ae991ed0-e51c-4804-b9c3-31e0152a6170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 12:45:34.346623 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:34.346590    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-j2w5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 12:45:34.346819 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:34.346645    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h" podUID="ae991ed0-e51c-4804-b9c3-31e0152a6170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 12:45:35.346785 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:35.346754    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-j2w5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 12:45:35.347180 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:35.346800    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h" podUID="ae991ed0-e51c-4804-b9c3-31e0152a6170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 12:45:36.347184 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:36.347148    4522 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-j2w5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 12:45:36.347569 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:36.347192    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h" podUID="ae991ed0-e51c-4804-b9c3-31e0152a6170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 12:45:37.874513 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:37.874482    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:37.874988 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:37.874776    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:37.875494 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:37.875471    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:45:46.892718 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:46.892636    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:51.135966 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:45:51.135926    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:45:54.102295 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:54.102260    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:54.186209 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:45:54.186180    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:01.152945 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.152912    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:01.253421 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.253387    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:01.476050 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.475984    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:01.572020 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.571995    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:01.696482 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.696453    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:01.763825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.763759    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"]
Oct 30 12:46:01.764281 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.764263    4522 cpu_manager.go:414] "containerMap: removing container" podUID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" containerName="cleanup"
Oct 30 12:46:01.764327 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.764285    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" containerName="cleanup"
Oct 30 12:46:01.764376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.764331    4522 cpu_manager.go:414] "containerMap: removing container" podUID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" containerName="cleanup"
Oct 30 12:46:01.764376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.764339    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" containerName="cleanup"
Oct 30 12:46:01.764465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.764453    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" containerName="cleanup"
Oct 30 12:46:01.764513 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.764468    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" containerName="cleanup"
Oct 30 12:46:01.771056 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.771037    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.788275 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.788250    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"]
Oct 30 12:46:01.797047 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.797027    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c1b2ed0a-openshift-default-dockercfg-4fdgl\""
Oct 30 12:46:01.809487 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.809462    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:01.831523 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.831500    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-credential-socket\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.831628 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.831532    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-envoy\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.831628 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.831551    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-istio-token\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.831713 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.831699    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f227556-8b10-40b3-aa88-cf825443e409-istiod-ca-cert\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.831748 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.831727    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f227556-8b10-40b3-aa88-cf825443e409-istio-podinfo\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.831781 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.831768    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-data\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.831818 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.831802    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6rf\" (UniqueName: \"kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-kube-api-access-bt6rf\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.831891 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.831874    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-certs\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.831929 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.831903    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-socket\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.885145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.885117    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"]
Oct 30 12:46:01.885563 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.885543    4522 cpu_manager.go:414] "containerMap: removing container" podUID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" containerName="cleanup"
Oct 30 12:46:01.885658 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.885564    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" containerName="cleanup"
Oct 30 12:46:01.885720 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.885662    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="683cf6fc-94c0-4d89-a18b-7b145e4bc5b5" containerName="cleanup"
Oct 30 12:46:01.889031 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.889011    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:01.904345 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.904326    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4da58cb4-openshift-default-dockercfg-rbsj7\""
Oct 30 12:46:01.911795 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.911772    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"]
Oct 30 12:46:01.932697 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.932673    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-data\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.932814 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.932722    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6rf\" (UniqueName: \"kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-kube-api-access-bt6rf\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.932814 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.932755    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-socket\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:01.932814 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.932784    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-certs\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.932814 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.932808    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-certs\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:01.933024 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.932845    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-istio-token\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.933024 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.932894    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istiod-ca-cert\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:01.933024 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.932939    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-credential-socket\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:01.933024 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.932970    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-socket\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.933024 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933000    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-credential-socket\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.933024 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933025    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-envoy\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.933315 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933080    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-data\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.933531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933507    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:01.933622 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933547    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-certs\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.933688 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933643    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gkjk\" (UniqueName: \"kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-kube-api-access-9gkjk\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:01.933777 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933761    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-podinfo\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:01.933838 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933804    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f227556-8b10-40b3-aa88-cf825443e409-istiod-ca-cert\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.933893 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933840    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-envoy\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:01.933893 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933844    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-socket\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.933893 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933863    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-data\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:01.934044 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933890    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f227556-8b10-40b3-aa88-cf825443e409-istio-podinfo\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.934044 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933895    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-credential-socket\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.934044 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.933915    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-token\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:01.934514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.934490    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f227556-8b10-40b3-aa88-cf825443e409-istiod-ca-cert\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.935314 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.935288    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-envoy\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.936070 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.936048    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f227556-8b10-40b3-aa88-cf825443e409-istio-podinfo\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.955289 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.955269    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6rf\" (UniqueName: \"kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-kube-api-access-bt6rf\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:01.959810 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:01.959789    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-istio-token\") pod \"e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") " pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:02.034893 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.034873    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-socket\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.035035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.034913    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-certs\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.035035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.034951    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istiod-ca-cert\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.035035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.034975    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-credential-socket\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.035208 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.035035    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gkjk\" (UniqueName: \"kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-kube-api-access-9gkjk\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.035208 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.035081    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-podinfo\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.035208 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.035122    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-envoy\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.035208 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.035139    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-data\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.035208 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.035153    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-token\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.035509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.035226    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-socket\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.035509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.035276    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-certs\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.035621 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.035584    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-data\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.035672 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.035614    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-credential-socket\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.035733 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.035703    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istiod-ca-cert\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.036030 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.036006    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:02.037159 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.037142    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-envoy\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.037624 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.037589    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-podinfo\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.057449 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.057425    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-token\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.058571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.058555    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gkjk\" (UniqueName: \"kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-kube-api-access-9gkjk\") pod \"e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") " pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.083452 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.083430    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:02.155504 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.155371    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:02.204552 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.204076    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:02.278480 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.278449    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:02.399432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.398790    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"]
Oct 30 12:46:02.401540 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:02.401510    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f227556_8b10_40b3_aa88_cf825443e409.slice/crio-1984963ca1b43373631c6b6e6e0213ba1076841cc60ff3ec1514eec30811c27c WatchSource:0}: Error finding container 1984963ca1b43373631c6b6e6e0213ba1076841cc60ff3ec1514eec30811c27c: Status 404 returned error can't find the container with id 1984963ca1b43373631c6b6e6e0213ba1076841cc60ff3ec1514eec30811c27c
Oct 30 12:46:02.403960 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.403918    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:02.404053 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.404012    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:02.404095 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.404055    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:02.518869 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.518834    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:02.558885 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.558774    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"]
Oct 30 12:46:02.565792 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:02.565741    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31d98d4_f212_43b5_8af6_e40a792bdcdf.slice/crio-1a8016b754a19dd24e6b33590e3ab1389a517df9c30c346c68f47ebe79e995ea WatchSource:0}: Error finding container 1a8016b754a19dd24e6b33590e3ab1389a517df9c30c346c68f47ebe79e995ea: Status 404 returned error can't find the container with id 1a8016b754a19dd24e6b33590e3ab1389a517df9c30c346c68f47ebe79e995ea
Oct 30 12:46:02.568566 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.568515    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:02.568686 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.568613    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:02.568686 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.568657    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:02.844778 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.844681    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" event={"ID":"d31d98d4-f212-43b5-8af6-e40a792bdcdf","Type":"ContainerStarted","Data":"7cc68a547ac657bee3768687515a54d8ae444ee3bab506bd2f25bc4a7a9453ad"}
Oct 30 12:46:02.844778 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.844747    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" event={"ID":"d31d98d4-f212-43b5-8af6-e40a792bdcdf","Type":"ContainerStarted","Data":"1a8016b754a19dd24e6b33590e3ab1389a517df9c30c346c68f47ebe79e995ea"}
Oct 30 12:46:02.846298 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.846271    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" event={"ID":"1f227556-8b10-40b3-aa88-cf825443e409","Type":"ContainerStarted","Data":"8cb0aaa0cf02bf785ec204aa7fa55002bdb32299b8b509464fb5c43f2f8ba69b"}
Oct 30 12:46:02.846441 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.846312    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" event={"ID":"1f227556-8b10-40b3-aa88-cf825443e409","Type":"ContainerStarted","Data":"1984963ca1b43373631c6b6e6e0213ba1076841cc60ff3ec1514eec30811c27c"}
Oct 30 12:46:02.886198 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.886153    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" podStartSLOduration=1.886139895 podStartE2EDuration="1.886139895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:02.884403483 +0000 UTC m=+606.699729973" watchObservedRunningTime="2026-10-30 12:46:02.886139895 +0000 UTC m=+606.701466430"
Oct 30 12:46:02.970525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:02.970500    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:03.085000 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:03.084956    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:03.089785 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:03.089752    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:03.129817 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:03.129734    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" podStartSLOduration=2.129721056 podStartE2EDuration="2.129721056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:02.929483196 +0000 UTC m=+606.744809684" watchObservedRunningTime="2026-10-30 12:46:03.129721056 +0000 UTC m=+606.945047544"
Oct 30 12:46:03.205462 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:03.205430    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:03.210113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:03.210090    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:03.418097 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:03.417985    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:03.519775 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:03.519747    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:03.851776 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:03.851734    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:03.851953 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:03.851800    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:03.852459 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:03.852425    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:03.852603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:03.852481    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:04.542563 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:04.542527    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:04.757099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:04.757068    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:04.873890 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:04.873828    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:04.974027 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:04.973998    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:05.300476 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:05.300446    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:05.400256 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:05.400231    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:06.089030 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.088998    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:06.121415 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.121388    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"]
Oct 30 12:46:06.125080 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.125060    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.140307 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.140290    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:46:06.154877 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.154855    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cdd5b3ae-serving-cert\""
Oct 30 12:46:06.155007 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.154992    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mwj6w\""
Oct 30 12:46:06.155074 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.155021    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 12:46:06.156129 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.155260    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:46:06.157769 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.157748    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"]
Oct 30 12:46:06.195711 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.195679    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:06.196782 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.196762    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c31a64e-d5e2-4883-a3b2-73631dbf5254-openshift-service-ca\") pod \"maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.196853 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.196791    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-maas-api-tls\") pod \"maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.196853 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.196827    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7274n\" (UniqueName: \"kubernetes.io/projected/3c31a64e-d5e2-4883-a3b2-73631dbf5254-kube-api-access-7274n\") pod \"maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.196948 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.196907    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-metrics-certs\") pod \"maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.246968 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.246922    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"]
Oct 30 12:46:06.250919 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.250892    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"
Oct 30 12:46:06.270146 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.269923    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"]
Oct 30 12:46:06.270146 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.269957    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cdd5b3ae\""
Oct 30 12:46:06.270456 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.270433    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cdd5b3ae-dockercfg-rkkmw\""
Oct 30 12:46:06.297890 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.297866    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c31a64e-d5e2-4883-a3b2-73631dbf5254-openshift-service-ca\") pod \"maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.297982 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.297908    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54813bab-e748-45cf-a700-23984fc19915-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4\" (UID: \"54813bab-e748-45cf-a700-23984fc19915\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"
Oct 30 12:46:06.297982 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.297932    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-maas-api-tls\") pod \"maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.297982 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.297952    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fh4q\" (UniqueName: \"kubernetes.io/projected/54813bab-e748-45cf-a700-23984fc19915-kube-api-access-9fh4q\") pod \"payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4\" (UID: \"54813bab-e748-45cf-a700-23984fc19915\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"
Oct 30 12:46:06.298162 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.298004    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7274n\" (UniqueName: \"kubernetes.io/projected/3c31a64e-d5e2-4883-a3b2-73631dbf5254-kube-api-access-7274n\") pod \"maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.298162 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:06.298097    4522 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-serving-cert: secret "maas-api-e2e-migrate-cdd5b3ae-serving-cert" not found
Oct 30 12:46:06.298162 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.298131    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-metrics-certs\") pod \"maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.298317 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:06.298168    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-maas-api-tls podName:3c31a64e-d5e2-4883-a3b2-73631dbf5254 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:06.79813091 +0000 UTC m=+610.613457389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-maas-api-tls") pod "maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x" (UID: "3c31a64e-d5e2-4883-a3b2-73631dbf5254") : secret "maas-api-e2e-migrate-cdd5b3ae-serving-cert" not found
Oct 30 12:46:06.298638 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.298620    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c31a64e-d5e2-4883-a3b2-73631dbf5254-openshift-service-ca\") pod \"maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.300662 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.300637    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-metrics-certs\") pod \"maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.302508 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.302489    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:06.318149 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.318131    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7274n\" (UniqueName: \"kubernetes.io/projected/3c31a64e-d5e2-4883-a3b2-73631dbf5254-kube-api-access-7274n\") pod \"maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.387117 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.387059    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"]
Oct 30 12:46:06.390657 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.390638    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"
Oct 30 12:46:06.399592 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.399567    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54813bab-e748-45cf-a700-23984fc19915-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4\" (UID: \"54813bab-e748-45cf-a700-23984fc19915\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"
Oct 30 12:46:06.399677 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.399627    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fh4q\" (UniqueName: \"kubernetes.io/projected/54813bab-e748-45cf-a700-23984fc19915-kube-api-access-9fh4q\") pod \"payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4\" (UID: \"54813bab-e748-45cf-a700-23984fc19915\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"
Oct 30 12:46:06.400272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.400256    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54813bab-e748-45cf-a700-23984fc19915-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4\" (UID: \"54813bab-e748-45cf-a700-23984fc19915\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"
Oct 30 12:46:06.409315 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.409293    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"]
Oct 30 12:46:06.418930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.418901    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fh4q\" (UniqueName: \"kubernetes.io/projected/54813bab-e748-45cf-a700-23984fc19915-kube-api-access-9fh4q\") pod \"payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4\" (UID: \"54813bab-e748-45cf-a700-23984fc19915\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"
Oct 30 12:46:06.468255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.468235    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:06.507429 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.507327    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d638c416-f301-4fbc-b33a-0e795ceb9a56-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d\" (UID: \"d638c416-f301-4fbc-b33a-0e795ceb9a56\") " pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"
Oct 30 12:46:06.507647 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.507583    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpwbd\" (UniqueName: \"kubernetes.io/projected/d638c416-f301-4fbc-b33a-0e795ceb9a56-kube-api-access-tpwbd\") pod \"payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d\" (UID: \"d638c416-f301-4fbc-b33a-0e795ceb9a56\") " pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"
Oct 30 12:46:06.568321 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.568291    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"
Oct 30 12:46:06.608663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.608633    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpwbd\" (UniqueName: \"kubernetes.io/projected/d638c416-f301-4fbc-b33a-0e795ceb9a56-kube-api-access-tpwbd\") pod \"payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d\" (UID: \"d638c416-f301-4fbc-b33a-0e795ceb9a56\") " pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"
Oct 30 12:46:06.608806 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.608745    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d638c416-f301-4fbc-b33a-0e795ceb9a56-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d\" (UID: \"d638c416-f301-4fbc-b33a-0e795ceb9a56\") " pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"
Oct 30 12:46:06.609437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.609420    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d638c416-f301-4fbc-b33a-0e795ceb9a56-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d\" (UID: \"d638c416-f301-4fbc-b33a-0e795ceb9a56\") " pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"
Oct 30 12:46:06.634114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.634056    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpwbd\" (UniqueName: \"kubernetes.io/projected/d638c416-f301-4fbc-b33a-0e795ceb9a56-kube-api-access-tpwbd\") pod \"payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d\" (UID: \"d638c416-f301-4fbc-b33a-0e795ceb9a56\") " pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"
Oct 30 12:46:06.703808 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.703738    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"
Oct 30 12:46:06.811056 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.811030    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-maas-api-tls\") pod \"maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.813261 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.813236    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-maas-api-tls\") pod \"maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:06.872343 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:06.872317    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54813bab_e748_45cf_a700_23984fc19915.slice/crio-e18c29343058021ba00c06d0c0f503629eca719bc06d799a95b057625a9c67a9 WatchSource:0}: Error finding container e18c29343058021ba00c06d0c0f503629eca719bc06d799a95b057625a9c67a9: Status 404 returned error can't find the container with id e18c29343058021ba00c06d0c0f503629eca719bc06d799a95b057625a9c67a9
Oct 30 12:46:06.872675 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.872654    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"]
Oct 30 12:46:06.997151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:06.997131    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"]
Oct 30 12:46:06.997920 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:06.997894    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd638c416_f301_4fbc_b33a_0e795ceb9a56.slice/crio-40e0f22c42da7e0a28711ae75bbed2ff599bf33072858a5088e81763107d22dc WatchSource:0}: Error finding container 40e0f22c42da7e0a28711ae75bbed2ff599bf33072858a5088e81763107d22dc: Status 404 returned error can't find the container with id 40e0f22c42da7e0a28711ae75bbed2ff599bf33072858a5088e81763107d22dc
Oct 30 12:46:07.044303 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.038678    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:07.181563 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.181536    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:07.294802 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.294779    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:07.350468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.350443    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"]
Oct 30 12:46:07.350786 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:07.350760    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c31a64e_d5e2_4883_a3b2_73631dbf5254.slice/crio-296404939d9bb378a4cfc1c5d91fbc6290436151e94d98a393a1325f9732ca8a WatchSource:0}: Error finding container 296404939d9bb378a4cfc1c5d91fbc6290436151e94d98a393a1325f9732ca8a: Status 404 returned error can't find the container with id 296404939d9bb378a4cfc1c5d91fbc6290436151e94d98a393a1325f9732ca8a
Oct 30 12:46:07.352412 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.352393    4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:46:07.873105 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.873065    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d" event={"ID":"d638c416-f301-4fbc-b33a-0e795ceb9a56","Type":"ContainerStarted","Data":"690b723d0a05efec50dbf7adad1fd65aaeca8b15f15d4f051ff56cfaeaf5ebe3"}
Oct 30 12:46:07.873288 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.873136    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"
Oct 30 12:46:07.873288 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.873160    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d" event={"ID":"d638c416-f301-4fbc-b33a-0e795ceb9a56","Type":"ContainerStarted","Data":"40e0f22c42da7e0a28711ae75bbed2ff599bf33072858a5088e81763107d22dc"}
Oct 30 12:46:07.875141 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.874710    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4" event={"ID":"54813bab-e748-45cf-a700-23984fc19915","Type":"ContainerStarted","Data":"448eca8fb4394b9b55a201b425ee543ed24caf92885b4c33935d38c303b2dbfb"}
Oct 30 12:46:07.875141 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.874772    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4" event={"ID":"54813bab-e748-45cf-a700-23984fc19915","Type":"ContainerStarted","Data":"e18c29343058021ba00c06d0c0f503629eca719bc06d799a95b057625a9c67a9"}
Oct 30 12:46:07.875141 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.874811    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"
Oct 30 12:46:07.881427 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.881398    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x" event={"ID":"3c31a64e-d5e2-4883-a3b2-73631dbf5254","Type":"ContainerStarted","Data":"296404939d9bb378a4cfc1c5d91fbc6290436151e94d98a393a1325f9732ca8a"}
Oct 30 12:46:07.916628 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.916591    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d" podStartSLOduration=1.916579009 podStartE2EDuration="1.916579009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:07.915242144 +0000 UTC m=+611.730568632" watchObservedRunningTime="2026-10-30 12:46:07.916579009 +0000 UTC m=+611.731905496"
Oct 30 12:46:07.935785 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.935753    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:07.959197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:07.959146    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4" podStartSLOduration=1.959128853 podStartE2EDuration="1.959128853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:07.956651581 +0000 UTC m=+611.771978069" watchObservedRunningTime="2026-10-30 12:46:07.959128853 +0000 UTC m=+611.774455343"
Oct 30 12:46:08.045996 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.045961    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:08.051051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.051029    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"]
Oct 30 12:46:08.054441 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.054427    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"
Oct 30 12:46:08.074478 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.074459    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"]
Oct 30 12:46:08.082529 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.082507    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"]
Oct 30 12:46:08.085857 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.085841    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.108637 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.108612    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4da58cb4-serving-cert\""
Oct 30 12:46:08.117444 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.117320    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"]
Oct 30 12:46:08.146370 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.146291    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-metrics-certs\") pod \"maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.146370 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.146341    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-maas-api-tls\") pod \"maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.146506 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.146458    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqqrn\" (UniqueName: \"kubernetes.io/projected/92a9d049-20c0-42fa-978b-169ed1c58b53-kube-api-access-zqqrn\") pod \"payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq\" (UID: \"92a9d049-20c0-42fa-978b-169ed1c58b53\") " pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"
Oct 30 12:46:08.146506 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.146494    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpq6z\" (UniqueName: \"kubernetes.io/projected/c9a3941e-fbf5-4ebe-a833-4b919078d46f-kube-api-access-mpq6z\") pod \"maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.146588 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.146519    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a3941e-fbf5-4ebe-a833-4b919078d46f-openshift-service-ca\") pod \"maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.146588 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.146580    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a9d049-20c0-42fa-978b-169ed1c58b53-plugins-config-volume\") pod \"payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq\" (UID: \"92a9d049-20c0-42fa-978b-169ed1c58b53\") " pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"
Oct 30 12:46:08.148919 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.148895    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4da58cb4\""
Oct 30 12:46:08.183004 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.182980    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:08.247972 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.247953    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a9d049-20c0-42fa-978b-169ed1c58b53-plugins-config-volume\") pod \"payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq\" (UID: \"92a9d049-20c0-42fa-978b-169ed1c58b53\") " pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"
Oct 30 12:46:08.248088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.247981    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-metrics-certs\") pod \"maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.248088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.248009    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-maas-api-tls\") pod \"maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.248088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.248063    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqqrn\" (UniqueName: \"kubernetes.io/projected/92a9d049-20c0-42fa-978b-169ed1c58b53-kube-api-access-zqqrn\") pod \"payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq\" (UID: \"92a9d049-20c0-42fa-978b-169ed1c58b53\") " pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"
Oct 30 12:46:08.248088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.248083    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpq6z\" (UniqueName: \"kubernetes.io/projected/c9a3941e-fbf5-4ebe-a833-4b919078d46f-kube-api-access-mpq6z\") pod \"maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.248408 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.248098    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a3941e-fbf5-4ebe-a833-4b919078d46f-openshift-service-ca\") pod \"maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.248891 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.248869    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a9d049-20c0-42fa-978b-169ed1c58b53-plugins-config-volume\") pod \"payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq\" (UID: \"92a9d049-20c0-42fa-978b-169ed1c58b53\") " pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"
Oct 30 12:46:08.249010 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.248990    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a3941e-fbf5-4ebe-a833-4b919078d46f-openshift-service-ca\") pod \"maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.250321 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.250298    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-metrics-certs\") pod \"maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.250416 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.250370    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-maas-api-tls\") pod \"maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.267089 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.267065    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqqrn\" (UniqueName: \"kubernetes.io/projected/92a9d049-20c0-42fa-978b-169ed1c58b53-kube-api-access-zqqrn\") pod \"payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq\" (UID: \"92a9d049-20c0-42fa-978b-169ed1c58b53\") " pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"
Oct 30 12:46:08.267448 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.267427    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpq6z\" (UniqueName: \"kubernetes.io/projected/c9a3941e-fbf5-4ebe-a833-4b919078d46f-kube-api-access-mpq6z\") pod \"maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.291890 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.291867    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:08.366407 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.366334    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"
Oct 30 12:46:08.402214 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.402142    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:08.414393 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.412617    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:08.695692 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:08.695665    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a9d049_20c0_42fa_978b_169ed1c58b53.slice/crio-0ef87311254b5bcc37fb3949bf0af1592375aa650bc0d0e60a9cf07b5857e372 WatchSource:0}: Error finding container 0ef87311254b5bcc37fb3949bf0af1592375aa650bc0d0e60a9cf07b5857e372: Status 404 returned error can't find the container with id 0ef87311254b5bcc37fb3949bf0af1592375aa650bc0d0e60a9cf07b5857e372
Oct 30 12:46:08.696049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.696027    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"]
Oct 30 12:46:08.730008 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:08.729985    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a3941e_fbf5_4ebe_a833_4b919078d46f.slice/crio-9a93e42533c2a491b006ef8b4639ec06bd5b4f09c67e92dec985ad31adb55919 WatchSource:0}: Error finding container 9a93e42533c2a491b006ef8b4639ec06bd5b4f09c67e92dec985ad31adb55919: Status 404 returned error can't find the container with id 9a93e42533c2a491b006ef8b4639ec06bd5b4f09c67e92dec985ad31adb55919
Oct 30 12:46:08.730235 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.730215    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"]
Oct 30 12:46:08.862172 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.862143    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:08.888839 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.888812    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x" event={"ID":"3c31a64e-d5e2-4883-a3b2-73631dbf5254","Type":"ContainerStarted","Data":"280ad5865879f4c177194e3d543cec39e266722f87b123a3b8fbd61afcbdf6ea"}
Oct 30 12:46:08.889023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.889008    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:08.890824 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.890790    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq" event={"ID":"92a9d049-20c0-42fa-978b-169ed1c58b53","Type":"ContainerStarted","Data":"3c05548f68281fa3475dbbc9f67583d03b2adb716508ec5b011770984c9b1f43"}
Oct 30 12:46:08.890972 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.890843    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"
Oct 30 12:46:08.890972 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.890867    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq" event={"ID":"92a9d049-20c0-42fa-978b-169ed1c58b53","Type":"ContainerStarted","Data":"0ef87311254b5bcc37fb3949bf0af1592375aa650bc0d0e60a9cf07b5857e372"}
Oct 30 12:46:08.892187 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.892142    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq" event={"ID":"c9a3941e-fbf5-4ebe-a833-4b919078d46f","Type":"ContainerStarted","Data":"9a93e42533c2a491b006ef8b4639ec06bd5b4f09c67e92dec985ad31adb55919"}
Oct 30 12:46:08.931626 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.931575    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x" podStartSLOduration=2.316679369 podStartE2EDuration="2.931563252s" totalImagesPullingTime="614.883883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:08.929552348 +0000 UTC m=+612.744878836" watchObservedRunningTime="2026-10-30 12:46:08.931563252 +0000 UTC m=+612.746889739"
Oct 30 12:46:08.969711 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.969645    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq" podStartSLOduration=0.969634926 podStartE2EDuration="969.634926ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:08.968455436 +0000 UTC m=+612.783781949" watchObservedRunningTime="2026-10-30 12:46:08.969634926 +0000 UTC m=+612.784961412"
Oct 30 12:46:08.978400 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:08.978376    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:09.485659 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:09.485628    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:09.583505 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:09.583473    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:09.730374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:09.730330    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:09.825990 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:09.825963    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:09.898818 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:09.898787    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq" event={"ID":"c9a3941e-fbf5-4ebe-a833-4b919078d46f","Type":"ContainerStarted","Data":"c78324734b06c9953b416fb47fe1ff8b55b4a11ed2c3a64bc9b62b031fe42d3e"}
Oct 30 12:46:10.014097 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:10.014064    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:10.115144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:10.115073    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:10.226222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:10.226162    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:10.339252 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:10.339204    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:10.461426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:10.456093    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:10.903127 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:10.903100    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:10.926292 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:10.926246    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq" podStartSLOduration=2.62424785 podStartE2EDuration="2.92623212s" totalImagesPullingTime="301.98427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:09.970695187 +0000 UTC m=+613.786021676" watchObservedRunningTime="2026-10-30 12:46:10.92623212 +0000 UTC m=+614.741558607"
Oct 30 12:46:10.927243 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:10.927222    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:46:11.801125 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:11.801081    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"]
Oct 30 12:46:11.812800 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:11.812769    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:11.848784 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:11.848551    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"]
Oct 30 12:46:11.854638 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:11.854582    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c1b2ed0a-serving-cert\""
Oct 30 12:46:11.904642 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:11.904607    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:11.905501 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:11.905477    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-metrics-certs\") pod \"maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:11.905562 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:11.905514    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a45d37fb-c68f-4347-b922-4096609dab99-openshift-service-ca\") pod \"maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:11.905639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:11.905628    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjkcl\" (UniqueName: \"kubernetes.io/projected/a45d37fb-c68f-4347-b922-4096609dab99-kube-api-access-kjkcl\") pod \"maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:11.905672 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:11.905655    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-maas-api-tls\") pod \"maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:12.005090 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:12.005052    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:12.006278 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:12.006257    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-metrics-certs\") pod \"maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:12.006333 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:12.006290    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a45d37fb-c68f-4347-b922-4096609dab99-openshift-service-ca\") pod \"maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:12.006490 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:12.006474    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjkcl\" (UniqueName: \"kubernetes.io/projected/a45d37fb-c68f-4347-b922-4096609dab99-kube-api-access-kjkcl\") pod \"maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:12.006546 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:12.006529    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-maas-api-tls\") pod \"maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:12.007092 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:12.007070    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a45d37fb-c68f-4347-b922-4096609dab99-openshift-service-ca\") pod \"maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:12.008702 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:12.008682    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-metrics-certs\") pod \"maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:12.008897 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:12.008874    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-maas-api-tls\") pod \"maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:12.045867 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:12.045834    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjkcl\" (UniqueName: \"kubernetes.io/projected/a45d37fb-c68f-4347-b922-4096609dab99-kube-api-access-kjkcl\") pod \"maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:12.167058 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:12.166977    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:12.493631 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:12.493596    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45d37fb_c68f_4347_b922_4096609dab99.slice/crio-a673cf117912c95680932cfb8bd2d81ca82e1be5692e1db23b21dbd93ff8f1da WatchSource:0}: Error finding container a673cf117912c95680932cfb8bd2d81ca82e1be5692e1db23b21dbd93ff8f1da: Status 404 returned error can't find the container with id a673cf117912c95680932cfb8bd2d81ca82e1be5692e1db23b21dbd93ff8f1da
Oct 30 12:46:12.495529 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:12.495489    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"]
Oct 30 12:46:12.914325 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:12.914284    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w" event={"ID":"a45d37fb-c68f-4347-b922-4096609dab99","Type":"ContainerStarted","Data":"a673cf117912c95680932cfb8bd2d81ca82e1be5692e1db23b21dbd93ff8f1da"}
Oct 30 12:46:13.236487 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:13.236411    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:13.436434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:13.436402    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:13.920293 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:13.920259    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w" event={"ID":"a45d37fb-c68f-4347-b922-4096609dab99","Type":"ContainerStarted","Data":"b8f77a5ae892cbe5a61b1cdeec040c9d7b68528ee2adc2ddd952a211b17cc019"}
Oct 30 12:46:13.920705 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:13.920385    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:13.964723 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:13.964674    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w" podStartSLOduration=2.544336113 podStartE2EDuration="2.96465955s" totalImagesPullingTime="420.323437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:13.962988042 +0000 UTC m=+617.778314533" watchObservedRunningTime="2026-10-30 12:46:13.96465955 +0000 UTC m=+617.779986036"
Oct 30 12:46:14.065024 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:14.064988    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:14.904940 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:14.904910    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:15.090217 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.090184    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:15.207557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.207482    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:15.217669 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.217641    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"]
Oct 30 12:46:15.221228 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.221214    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"
Oct 30 12:46:15.241401 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.241146    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-48623dd7\""
Oct 30 12:46:15.241401 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.241150    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-48623dd7-dockercfg-dx66b\""
Oct 30 12:46:15.243567 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.243547    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"]
Oct 30 12:46:15.360267 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.360224    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7f9517-46e8-4254-bb3b-a6995326049a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s\" (UID: \"9b7f9517-46e8-4254-bb3b-a6995326049a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"
Oct 30 12:46:15.360663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.360639    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7v7r\" (UniqueName: \"kubernetes.io/projected/9b7f9517-46e8-4254-bb3b-a6995326049a-kube-api-access-g7v7r\") pod \"payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s\" (UID: \"9b7f9517-46e8-4254-bb3b-a6995326049a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"
Oct 30 12:46:15.461328 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.461237    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7v7r\" (UniqueName: \"kubernetes.io/projected/9b7f9517-46e8-4254-bb3b-a6995326049a-kube-api-access-g7v7r\") pod \"payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s\" (UID: \"9b7f9517-46e8-4254-bb3b-a6995326049a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"
Oct 30 12:46:15.461539 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.461327    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7f9517-46e8-4254-bb3b-a6995326049a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s\" (UID: \"9b7f9517-46e8-4254-bb3b-a6995326049a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"
Oct 30 12:46:15.462060 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.462037    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7f9517-46e8-4254-bb3b-a6995326049a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s\" (UID: \"9b7f9517-46e8-4254-bb3b-a6995326049a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"
Oct 30 12:46:15.495333 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.495303    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7v7r\" (UniqueName: \"kubernetes.io/projected/9b7f9517-46e8-4254-bb3b-a6995326049a-kube-api-access-g7v7r\") pod \"payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s\" (UID: \"9b7f9517-46e8-4254-bb3b-a6995326049a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"
Oct 30 12:46:15.534680 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.534655    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"
Oct 30 12:46:15.853380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.853340    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"]
Oct 30 12:46:15.854757 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:15.854732    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7f9517_46e8_4254_bb3b_a6995326049a.slice/crio-33af60508310d397f55e381f69e90ad4a478d686d3c5a387372df51abbc1d5e4 WatchSource:0}: Error finding container 33af60508310d397f55e381f69e90ad4a478d686d3c5a387372df51abbc1d5e4: Status 404 returned error can't find the container with id 33af60508310d397f55e381f69e90ad4a478d686d3c5a387372df51abbc1d5e4
Oct 30 12:46:15.929504 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:15.929471    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s" event={"ID":"9b7f9517-46e8-4254-bb3b-a6995326049a","Type":"ContainerStarted","Data":"33af60508310d397f55e381f69e90ad4a478d686d3c5a387372df51abbc1d5e4"}
Oct 30 12:46:16.014668 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:16.014627    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:16.913520 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:16.913492    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:16.935778 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:16.935748    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s" event={"ID":"9b7f9517-46e8-4254-bb3b-a6995326049a","Type":"ContainerStarted","Data":"4e55e36058cd20f0a91e4e11141884c7ed4d83af5c1888699999028562d80100"}
Oct 30 12:46:16.935945 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:16.935863    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"
Oct 30 12:46:16.998376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:16.997768    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s" podStartSLOduration=1.997750978 podStartE2EDuration="1.997750978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:16.99571432 +0000 UTC m=+620.811040810" watchObservedRunningTime="2026-10-30 12:46:16.997750978 +0000 UTC m=+620.813077471"
Oct 30 12:46:17.812877 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:17.812841    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:18.122114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:18.122038    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:46:18.894827 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:18.894798    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"
Oct 30 12:46:18.895041 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:18.894873    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"
Oct 30 12:46:19.694265 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:19.694235    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:19.894552 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:19.894501    4522 patch_prober.go:29] interesting pod/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:46:19.894748 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:19.894583    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:46:19.901901 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:19.901878    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"
Oct 30 12:46:19.929269 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:19.929245    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:21.133150 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:21.133074    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:46:21.690376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:21.689762    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:21.789039 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:21.788805    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:21.961264 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:21.961183    4522 generic.go:356] "Generic (PLEG): container finished" podID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerID="7cc68a547ac657bee3768687515a54d8ae444ee3bab506bd2f25bc4a7a9453ad" exitCode=0
Oct 30 12:46:21.961264 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:21.961229    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" event={"ID":"d31d98d4-f212-43b5-8af6-e40a792bdcdf","Type":"ContainerDied","Data":"7cc68a547ac657bee3768687515a54d8ae444ee3bab506bd2f25bc4a7a9453ad"}
Oct 30 12:46:21.961642 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:21.961627    4522 scope.go:122] "RemoveContainer" containerID="7cc68a547ac657bee3768687515a54d8ae444ee3bab506bd2f25bc4a7a9453ad"
Oct 30 12:46:21.962961 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:21.962914    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:21.963021 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:21.962998    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:21.963057 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:21.963029    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:22.975379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:22.974185    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" event={"ID":"d31d98d4-f212-43b5-8af6-e40a792bdcdf","Type":"ContainerStarted","Data":"2b76a50affa3e4ebfcfa072f51a73b0e34e9384be6a815b5ed56a671f171cf48"}
Oct 30 12:46:23.205545 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:23.205502    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:23.206198 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:23.206158    4522 patch_prober.go:29] interesting pod/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 12:46:23.206458 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:23.206419    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 12:46:23.520650 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:23.519879    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:24.207752 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:24.207718    4522 patch_prober.go:29] interesting pod/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 12:46:24.208344 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:24.208316    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 12:46:24.254436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:24.252609    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:46:25.217379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:25.209577    4522 patch_prober.go:29] interesting pod/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 12:46:25.217379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:25.209636    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 12:46:25.360176 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:25.360137    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:26.205100 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:26.205053    4522 patch_prober.go:29] interesting pod/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 12:46:26.205299 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:26.205125    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 12:46:27.117091 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:27.117059    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:27.208204 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:27.206848    4522 patch_prober.go:29] interesting pod/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 12:46:27.208204 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:27.206915    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 12:46:27.943682 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:27.943644    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"
Oct 30 12:46:28.488392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:28.488346    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:28.490930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:28.488764    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:28.490930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:28.489479    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:29.048085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:29.048054    4522 generic.go:356] "Generic (PLEG): container finished" podID="1f227556-8b10-40b3-aa88-cf825443e409" containerID="8cb0aaa0cf02bf785ec204aa7fa55002bdb32299b8b509464fb5c43f2f8ba69b" exitCode=0
Oct 30 12:46:29.048251 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:29.048123    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" event={"ID":"1f227556-8b10-40b3-aa88-cf825443e409","Type":"ContainerDied","Data":"8cb0aaa0cf02bf785ec204aa7fa55002bdb32299b8b509464fb5c43f2f8ba69b"}
Oct 30 12:46:29.048576 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:29.048564    4522 scope.go:122] "RemoveContainer" containerID="8cb0aaa0cf02bf785ec204aa7fa55002bdb32299b8b509464fb5c43f2f8ba69b"
Oct 30 12:46:29.049711 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:29.049685    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:29.049777 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:29.049754    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:29.049816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:29.049785    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:29.058521 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:29.058494    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:30.054839 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:30.054808    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" event={"ID":"1f227556-8b10-40b3-aa88-cf825443e409","Type":"ContainerStarted","Data":"1acde284caca1c355f23bf973cd1506c3ad2fa211dac092de421fd942741a2fc"}
Oct 30 12:46:30.089375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:30.088855    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:30.092372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:30.091518    4522 patch_prober.go:29] interesting pod/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 12:46:30.092372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:30.091622    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" podUID="1f227556-8b10-40b3-aa88-cf825443e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 12:46:30.997922 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:30.997879    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"]
Oct 30 12:46:31.023085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:31.023056    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"]
Oct 30 12:46:31.045240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:31.045204    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:31.100808 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:31.100775    4522 patch_prober.go:29] interesting pod/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 12:46:31.101279 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:31.100833    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" podUID="1f227556-8b10-40b3-aa88-cf825443e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 12:46:31.144873 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:31.144836    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:31.248447 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:31.248297    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:32.086823 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:32.086790    4522 patch_prober.go:29] interesting pod/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 12:46:32.087142 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:32.087097    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" podUID="1f227556-8b10-40b3-aa88-cf825443e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 12:46:32.899986 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:32.899949    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:33.088379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:33.088016    4522 patch_prober.go:29] interesting pod/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 12:46:33.088379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:33.088072    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" podUID="1f227556-8b10-40b3-aa88-cf825443e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 12:46:33.851976 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:33.851944    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:34.092500 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:34.092464    4522 patch_prober.go:29] interesting pod/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 12:46:34.092901 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:34.092512    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" podUID="1f227556-8b10-40b3-aa88-cf825443e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 12:46:34.922293 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:34.922256    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:35.084808 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:35.084770    4522 patch_prober.go:29] interesting pod/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 12:46:35.084979 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:35.084839    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" podUID="1f227556-8b10-40b3-aa88-cf825443e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 12:46:35.861014 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:35.860980    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"]
Oct 30 12:46:35.988309 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:35.988270    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:35.999174 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:35.999141    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d" podUID="d638c416-f301-4fbc-b33a-0e795ceb9a56" containerName="payload-processing" containerID="cri-o://690b723d0a05efec50dbf7adad1fd65aaeca8b15f15d4f051ff56cfaeaf5ebe3" gracePeriod=25
Oct 30 12:46:36.023626 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.023591    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4" podUID="54813bab-e748-45cf-a700-23984fc19915" containerName="payload-pre-processing" containerID="cri-o://448eca8fb4394b9b55a201b425ee543ed24caf92885b4c33935d38c303b2dbfb" gracePeriod=25
Oct 30 12:46:36.117043 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.116934    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:36.125583 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.125541    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:36.128399 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.128292    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:36.291910 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.291884    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"
Oct 30 12:46:36.329595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.329099    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54813bab-e748-45cf-a700-23984fc19915-kube-api-access-9fh4q\" (UniqueName: \"kubernetes.io/projected/54813bab-e748-45cf-a700-23984fc19915-kube-api-access-9fh4q\") pod \"54813bab-e748-45cf-a700-23984fc19915\" (UID: \"54813bab-e748-45cf-a700-23984fc19915\") "
Oct 30 12:46:36.329595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.329187    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54813bab-e748-45cf-a700-23984fc19915-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54813bab-e748-45cf-a700-23984fc19915-plugins-config-volume\") pod \"54813bab-e748-45cf-a700-23984fc19915\" (UID: \"54813bab-e748-45cf-a700-23984fc19915\") "
Oct 30 12:46:36.329794 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.329710    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54813bab-e748-45cf-a700-23984fc19915-plugins-config-volume" pod "54813bab-e748-45cf-a700-23984fc19915" (UID: "54813bab-e748-45cf-a700-23984fc19915"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:36.332211 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.332184    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54813bab-e748-45cf-a700-23984fc19915-kube-api-access-9fh4q" pod "54813bab-e748-45cf-a700-23984fc19915" (UID: "54813bab-e748-45cf-a700-23984fc19915"). InnerVolumeSpecName "kube-api-access-9fh4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:36.334809 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.334781    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"
Oct 30 12:46:36.430921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.430819    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d638c416-f301-4fbc-b33a-0e795ceb9a56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d638c416-f301-4fbc-b33a-0e795ceb9a56-plugins-config-volume\") pod \"d638c416-f301-4fbc-b33a-0e795ceb9a56\" (UID: \"d638c416-f301-4fbc-b33a-0e795ceb9a56\") "
Oct 30 12:46:36.431091 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.431019    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d638c416-f301-4fbc-b33a-0e795ceb9a56-kube-api-access-tpwbd\" (UniqueName: \"kubernetes.io/projected/d638c416-f301-4fbc-b33a-0e795ceb9a56-kube-api-access-tpwbd\") pod \"d638c416-f301-4fbc-b33a-0e795ceb9a56\" (UID: \"d638c416-f301-4fbc-b33a-0e795ceb9a56\") "
Oct 30 12:46:36.432394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.431682    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54813bab-e748-45cf-a700-23984fc19915-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:36.432394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.431716    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fh4q\" (UniqueName: \"kubernetes.io/projected/54813bab-e748-45cf-a700-23984fc19915-kube-api-access-9fh4q\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:36.432394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.431820    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d638c416-f301-4fbc-b33a-0e795ceb9a56-plugins-config-volume" pod "d638c416-f301-4fbc-b33a-0e795ceb9a56" (UID: "d638c416-f301-4fbc-b33a-0e795ceb9a56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:36.434863 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.434833    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d638c416-f301-4fbc-b33a-0e795ceb9a56-kube-api-access-tpwbd" pod "d638c416-f301-4fbc-b33a-0e795ceb9a56" (UID: "d638c416-f301-4fbc-b33a-0e795ceb9a56"). InnerVolumeSpecName "kube-api-access-tpwbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:36.532535 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.532504    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpwbd\" (UniqueName: \"kubernetes.io/projected/d638c416-f301-4fbc-b33a-0e795ceb9a56-kube-api-access-tpwbd\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:36.532535 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.532531    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d638c416-f301-4fbc-b33a-0e795ceb9a56-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:36.774076 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:36.774044    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:37.137591 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.137507    4522 generic.go:356] "Generic (PLEG): container finished" podID="d638c416-f301-4fbc-b33a-0e795ceb9a56" containerID="690b723d0a05efec50dbf7adad1fd65aaeca8b15f15d4f051ff56cfaeaf5ebe3" exitCode=0
Oct 30 12:46:37.137591 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.137560    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"
Oct 30 12:46:37.138108 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.137597    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d" event={"ID":"d638c416-f301-4fbc-b33a-0e795ceb9a56","Type":"ContainerDied","Data":"690b723d0a05efec50dbf7adad1fd65aaeca8b15f15d4f051ff56cfaeaf5ebe3"}
Oct 30 12:46:37.138108 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.137695    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d" event={"ID":"d638c416-f301-4fbc-b33a-0e795ceb9a56","Type":"ContainerDied","Data":"40e0f22c42da7e0a28711ae75bbed2ff599bf33072858a5088e81763107d22dc"}
Oct 30 12:46:37.138108 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.137731    4522 scope.go:122] "RemoveContainer" containerID="690b723d0a05efec50dbf7adad1fd65aaeca8b15f15d4f051ff56cfaeaf5ebe3"
Oct 30 12:46:37.139123 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.139096    4522 generic.go:356] "Generic (PLEG): container finished" podID="54813bab-e748-45cf-a700-23984fc19915" containerID="448eca8fb4394b9b55a201b425ee543ed24caf92885b4c33935d38c303b2dbfb" exitCode=0
Oct 30 12:46:37.139207 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.139167    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"
Oct 30 12:46:37.139207 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.139193    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4" event={"ID":"54813bab-e748-45cf-a700-23984fc19915","Type":"ContainerDied","Data":"448eca8fb4394b9b55a201b425ee543ed24caf92885b4c33935d38c303b2dbfb"}
Oct 30 12:46:37.139288 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.139235    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4" event={"ID":"54813bab-e748-45cf-a700-23984fc19915","Type":"ContainerDied","Data":"e18c29343058021ba00c06d0c0f503629eca719bc06d799a95b057625a9c67a9"}
Oct 30 12:46:37.148067 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.148050    4522 scope.go:122] "RemoveContainer" containerID="690b723d0a05efec50dbf7adad1fd65aaeca8b15f15d4f051ff56cfaeaf5ebe3"
Oct 30 12:46:37.148321 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:37.148303    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690b723d0a05efec50dbf7adad1fd65aaeca8b15f15d4f051ff56cfaeaf5ebe3\": container with ID starting with 690b723d0a05efec50dbf7adad1fd65aaeca8b15f15d4f051ff56cfaeaf5ebe3 not found: ID does not exist" containerID="690b723d0a05efec50dbf7adad1fd65aaeca8b15f15d4f051ff56cfaeaf5ebe3"
Oct 30 12:46:37.148379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.148330    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690b723d0a05efec50dbf7adad1fd65aaeca8b15f15d4f051ff56cfaeaf5ebe3"} err="failed to get container status \"690b723d0a05efec50dbf7adad1fd65aaeca8b15f15d4f051ff56cfaeaf5ebe3\": rpc error: code = NotFound desc = could not find container \"690b723d0a05efec50dbf7adad1fd65aaeca8b15f15d4f051ff56cfaeaf5ebe3\": container with ID starting with 690b723d0a05efec50dbf7adad1fd65aaeca8b15f15d4f051ff56cfaeaf5ebe3 not found: ID does not exist"
Oct 30 12:46:37.148379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.148347    4522 scope.go:122] "RemoveContainer" containerID="448eca8fb4394b9b55a201b425ee543ed24caf92885b4c33935d38c303b2dbfb"
Oct 30 12:46:37.155787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.155771    4522 scope.go:122] "RemoveContainer" containerID="448eca8fb4394b9b55a201b425ee543ed24caf92885b4c33935d38c303b2dbfb"
Oct 30 12:46:37.156038 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:37.156019    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448eca8fb4394b9b55a201b425ee543ed24caf92885b4c33935d38c303b2dbfb\": container with ID starting with 448eca8fb4394b9b55a201b425ee543ed24caf92885b4c33935d38c303b2dbfb not found: ID does not exist" containerID="448eca8fb4394b9b55a201b425ee543ed24caf92885b4c33935d38c303b2dbfb"
Oct 30 12:46:37.156086 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.156045    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448eca8fb4394b9b55a201b425ee543ed24caf92885b4c33935d38c303b2dbfb"} err="failed to get container status \"448eca8fb4394b9b55a201b425ee543ed24caf92885b4c33935d38c303b2dbfb\": rpc error: code = NotFound desc = could not find container \"448eca8fb4394b9b55a201b425ee543ed24caf92885b4c33935d38c303b2dbfb\": container with ID starting with 448eca8fb4394b9b55a201b425ee543ed24caf92885b4c33935d38c303b2dbfb not found: ID does not exist"
Oct 30 12:46:37.185523 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.185492    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"]
Oct 30 12:46:37.197414 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.197379    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cdd5b3ae-5c9679df78-hn25d"]
Oct 30 12:46:37.230520 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.230489    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"]
Oct 30 12:46:37.234708 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.234685    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cdd5b3ae-db46c6579-6slh4"]
Oct 30 12:46:37.407534 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.407441    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:37.524127 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:37.524053    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:38.306686 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.306651    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"]
Oct 30 12:46:38.307498 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.306892    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x" podUID="3c31a64e-d5e2-4883-a3b2-73631dbf5254" containerName="maas-api" containerID="cri-o://280ad5865879f4c177194e3d543cec39e266722f87b123a3b8fbd61afcbdf6ea" gracePeriod=30
Oct 30 12:46:38.335784 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.335754    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54813bab-e748-45cf-a700-23984fc19915" path="/var/lib/kubelet/pods/54813bab-e748-45cf-a700-23984fc19915/volumes"
Oct 30 12:46:38.336251 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.336231    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d638c416-f301-4fbc-b33a-0e795ceb9a56" path="/var/lib/kubelet/pods/d638c416-f301-4fbc-b33a-0e795ceb9a56/volumes"
Oct 30 12:46:38.362594 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.362572    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:38.590527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.590504    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:38.674084 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.674059    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"]
Oct 30 12:46:38.674250 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.674230    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq" podUID="c9a3941e-fbf5-4ebe-a833-4b919078d46f" containerName="maas-api" containerID="cri-o://c78324734b06c9953b416fb47fe1ff8b55b4a11ed2c3a64bc9b62b031fe42d3e" gracePeriod=30
Oct 30 12:46:38.677840 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.677818    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c31a64e-d5e2-4883-a3b2-73631dbf5254-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c31a64e-d5e2-4883-a3b2-73631dbf5254-openshift-service-ca\") pod \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") "
Oct 30 12:46:38.677963 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.677921    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-maas-api-tls\") pod \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") "
Oct 30 12:46:38.678067 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.677979    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c31a64e-d5e2-4883-a3b2-73631dbf5254-kube-api-access-7274n\" (UniqueName: \"kubernetes.io/projected/3c31a64e-d5e2-4883-a3b2-73631dbf5254-kube-api-access-7274n\") pod \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") "
Oct 30 12:46:38.678067 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.678019    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-metrics-certs\") pod \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\" (UID: \"3c31a64e-d5e2-4883-a3b2-73631dbf5254\") "
Oct 30 12:46:38.678222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.678198    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c31a64e-d5e2-4883-a3b2-73631dbf5254-openshift-service-ca" pod "3c31a64e-d5e2-4883-a3b2-73631dbf5254" (UID: "3c31a64e-d5e2-4883-a3b2-73631dbf5254"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:38.678423 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.678397    4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c31a64e-d5e2-4883-a3b2-73631dbf5254-openshift-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:38.681799 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.681776    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c31a64e-d5e2-4883-a3b2-73631dbf5254-kube-api-access-7274n" pod "3c31a64e-d5e2-4883-a3b2-73631dbf5254" (UID: "3c31a64e-d5e2-4883-a3b2-73631dbf5254"). InnerVolumeSpecName "kube-api-access-7274n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:38.681799 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.681789    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-maas-api-tls" pod "3c31a64e-d5e2-4883-a3b2-73631dbf5254" (UID: "3c31a64e-d5e2-4883-a3b2-73631dbf5254"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:38.682342 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.680397    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-metrics-certs" pod "3c31a64e-d5e2-4883-a3b2-73631dbf5254" (UID: "3c31a64e-d5e2-4883-a3b2-73631dbf5254"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:38.742272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.742246    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:38.779926 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.779898    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-maas-api-tls\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:38.779926 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.779922    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7274n\" (UniqueName: \"kubernetes.io/projected/3c31a64e-d5e2-4883-a3b2-73631dbf5254-kube-api-access-7274n\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:38.779926 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.779932    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c31a64e-d5e2-4883-a3b2-73631dbf5254-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:38.901514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.901340    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:38.931322 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.931299    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:38.982128 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.982105    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a3941e-fbf5-4ebe-a833-4b919078d46f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a3941e-fbf5-4ebe-a833-4b919078d46f-openshift-service-ca\") pod \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") "
Oct 30 12:46:38.982255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.982149    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-maas-api-tls\") pod \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") "
Oct 30 12:46:38.982255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.982168    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a3941e-fbf5-4ebe-a833-4b919078d46f-kube-api-access-mpq6z\" (UniqueName: \"kubernetes.io/projected/c9a3941e-fbf5-4ebe-a833-4b919078d46f-kube-api-access-mpq6z\") pod \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") "
Oct 30 12:46:38.982255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.982236    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-metrics-certs\") pod \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\" (UID: \"c9a3941e-fbf5-4ebe-a833-4b919078d46f\") "
Oct 30 12:46:38.982583 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.982554    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a3941e-fbf5-4ebe-a833-4b919078d46f-openshift-service-ca" pod "c9a3941e-fbf5-4ebe-a833-4b919078d46f" (UID: "c9a3941e-fbf5-4ebe-a833-4b919078d46f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:38.984318 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.984290    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-metrics-certs" pod "c9a3941e-fbf5-4ebe-a833-4b919078d46f" (UID: "c9a3941e-fbf5-4ebe-a833-4b919078d46f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:38.984318 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.984311    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-maas-api-tls" pod "c9a3941e-fbf5-4ebe-a833-4b919078d46f" (UID: "c9a3941e-fbf5-4ebe-a833-4b919078d46f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:38.984492 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:38.984341    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a3941e-fbf5-4ebe-a833-4b919078d46f-kube-api-access-mpq6z" pod "c9a3941e-fbf5-4ebe-a833-4b919078d46f" (UID: "c9a3941e-fbf5-4ebe-a833-4b919078d46f"). InnerVolumeSpecName "kube-api-access-mpq6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:39.032365 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.032329    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:39.083819 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.083795    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:39.083819 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.083817    4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a3941e-fbf5-4ebe-a833-4b919078d46f-openshift-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:39.083946 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.083826    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a3941e-fbf5-4ebe-a833-4b919078d46f-maas-api-tls\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:39.083946 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.083838    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpq6z\" (UniqueName: \"kubernetes.io/projected/c9a3941e-fbf5-4ebe-a833-4b919078d46f-kube-api-access-mpq6z\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:39.197436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.194072    4522 generic.go:356] "Generic (PLEG): container finished" podID="c9a3941e-fbf5-4ebe-a833-4b919078d46f" containerID="c78324734b06c9953b416fb47fe1ff8b55b4a11ed2c3a64bc9b62b031fe42d3e" exitCode=0
Oct 30 12:46:39.197436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.194164    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq" event={"ID":"c9a3941e-fbf5-4ebe-a833-4b919078d46f","Type":"ContainerDied","Data":"c78324734b06c9953b416fb47fe1ff8b55b4a11ed2c3a64bc9b62b031fe42d3e"}
Oct 30 12:46:39.197436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.194208    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq" event={"ID":"c9a3941e-fbf5-4ebe-a833-4b919078d46f","Type":"ContainerDied","Data":"9a93e42533c2a491b006ef8b4639ec06bd5b4f09c67e92dec985ad31adb55919"}
Oct 30 12:46:39.197436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.194234    4522 scope.go:122] "RemoveContainer" containerID="c78324734b06c9953b416fb47fe1ff8b55b4a11ed2c3a64bc9b62b031fe42d3e"
Oct 30 12:46:39.197436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.194334    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"
Oct 30 12:46:39.227144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.226268    4522 generic.go:356] "Generic (PLEG): container finished" podID="3c31a64e-d5e2-4883-a3b2-73631dbf5254" containerID="280ad5865879f4c177194e3d543cec39e266722f87b123a3b8fbd61afcbdf6ea" exitCode=0
Oct 30 12:46:39.227144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.226318    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x" event={"ID":"3c31a64e-d5e2-4883-a3b2-73631dbf5254","Type":"ContainerDied","Data":"280ad5865879f4c177194e3d543cec39e266722f87b123a3b8fbd61afcbdf6ea"}
Oct 30 12:46:39.227144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.226397    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x" event={"ID":"3c31a64e-d5e2-4883-a3b2-73631dbf5254","Type":"ContainerDied","Data":"296404939d9bb378a4cfc1c5d91fbc6290436151e94d98a393a1325f9732ca8a"}
Oct 30 12:46:39.227144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.226474    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"
Oct 30 12:46:39.251676 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.251646    4522 scope.go:122] "RemoveContainer" containerID="c78324734b06c9953b416fb47fe1ff8b55b4a11ed2c3a64bc9b62b031fe42d3e"
Oct 30 12:46:39.252161 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:39.252134    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78324734b06c9953b416fb47fe1ff8b55b4a11ed2c3a64bc9b62b031fe42d3e\": container with ID starting with c78324734b06c9953b416fb47fe1ff8b55b4a11ed2c3a64bc9b62b031fe42d3e not found: ID does not exist" containerID="c78324734b06c9953b416fb47fe1ff8b55b4a11ed2c3a64bc9b62b031fe42d3e"
Oct 30 12:46:39.252259 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.252171    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78324734b06c9953b416fb47fe1ff8b55b4a11ed2c3a64bc9b62b031fe42d3e"} err="failed to get container status \"c78324734b06c9953b416fb47fe1ff8b55b4a11ed2c3a64bc9b62b031fe42d3e\": rpc error: code = NotFound desc = could not find container \"c78324734b06c9953b416fb47fe1ff8b55b4a11ed2c3a64bc9b62b031fe42d3e\": container with ID starting with c78324734b06c9953b416fb47fe1ff8b55b4a11ed2c3a64bc9b62b031fe42d3e not found: ID does not exist"
Oct 30 12:46:39.252259 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.252191    4522 scope.go:122] "RemoveContainer" containerID="280ad5865879f4c177194e3d543cec39e266722f87b123a3b8fbd61afcbdf6ea"
Oct 30 12:46:39.262616 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.262595    4522 scope.go:122] "RemoveContainer" containerID="280ad5865879f4c177194e3d543cec39e266722f87b123a3b8fbd61afcbdf6ea"
Oct 30 12:46:39.262916 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:39.262884    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280ad5865879f4c177194e3d543cec39e266722f87b123a3b8fbd61afcbdf6ea\": container with ID starting with 280ad5865879f4c177194e3d543cec39e266722f87b123a3b8fbd61afcbdf6ea not found: ID does not exist" containerID="280ad5865879f4c177194e3d543cec39e266722f87b123a3b8fbd61afcbdf6ea"
Oct 30 12:46:39.263033 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.262924    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280ad5865879f4c177194e3d543cec39e266722f87b123a3b8fbd61afcbdf6ea"} err="failed to get container status \"280ad5865879f4c177194e3d543cec39e266722f87b123a3b8fbd61afcbdf6ea\": rpc error: code = NotFound desc = could not find container \"280ad5865879f4c177194e3d543cec39e266722f87b123a3b8fbd61afcbdf6ea\": container with ID starting with 280ad5865879f4c177194e3d543cec39e266722f87b123a3b8fbd61afcbdf6ea not found: ID does not exist"
Oct 30 12:46:39.267104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.267076    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:39.328642 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.328619    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"]
Oct 30 12:46:39.347890 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.347866    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4da58cb4-69d85c894c-ghncq"]
Oct 30 12:46:39.388559 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.388531    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"]
Oct 30 12:46:39.393656 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.393635    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cdd5b3ae-779985cc9d-pgh7x"]
Oct 30 12:46:39.427281 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.427257    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:39.531323 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.531297    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:39.780163 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.780130    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"]
Oct 30 12:46:39.780400 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.780373    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy" containerID="cri-o://2b76a50affa3e4ebfcfa072f51a73b0e34e9384be6a815b5ed56a671f171cf48" gracePeriod=30
Oct 30 12:46:39.804605 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.804548    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:39.935036 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:39.934994    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:40.039138 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.039102    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:40.149202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.149116    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:40.332765 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.332731    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c31a64e-d5e2-4883-a3b2-73631dbf5254" path="/var/lib/kubelet/pods/3c31a64e-d5e2-4883-a3b2-73631dbf5254/volumes"
Oct 30 12:46:40.333396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.333375    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9a3941e-fbf5-4ebe-a833-4b919078d46f" path="/var/lib/kubelet/pods/c9a3941e-fbf5-4ebe-a833-4b919078d46f/volumes"
Oct 30 12:46:40.530516 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.530486    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"]
Oct 30 12:46:40.530930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.530911    4522 cpu_manager.go:414] "containerMap: removing container" podUID="d638c416-f301-4fbc-b33a-0e795ceb9a56" containerName="payload-processing"
Oct 30 12:46:40.531029 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.530931    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d638c416-f301-4fbc-b33a-0e795ceb9a56" containerName="payload-processing"
Oct 30 12:46:40.531029 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.530947    4522 cpu_manager.go:414] "containerMap: removing container" podUID="c9a3941e-fbf5-4ebe-a833-4b919078d46f" containerName="maas-api"
Oct 30 12:46:40.531029 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.530954    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a3941e-fbf5-4ebe-a833-4b919078d46f" containerName="maas-api"
Oct 30 12:46:40.531029 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.530996    4522 cpu_manager.go:414] "containerMap: removing container" podUID="54813bab-e748-45cf-a700-23984fc19915" containerName="payload-pre-processing"
Oct 30 12:46:40.531029 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.531005    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54813bab-e748-45cf-a700-23984fc19915" containerName="payload-pre-processing"
Oct 30 12:46:40.531029 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.531016    4522 cpu_manager.go:414] "containerMap: removing container" podUID="3c31a64e-d5e2-4883-a3b2-73631dbf5254" containerName="maas-api"
Oct 30 12:46:40.531029 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.531023    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c31a64e-d5e2-4883-a3b2-73631dbf5254" containerName="maas-api"
Oct 30 12:46:40.531400 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.531124    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d638c416-f301-4fbc-b33a-0e795ceb9a56" containerName="payload-processing"
Oct 30 12:46:40.531400 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.531135    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a3941e-fbf5-4ebe-a833-4b919078d46f" containerName="maas-api"
Oct 30 12:46:40.531400 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.531151    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="54813bab-e748-45cf-a700-23984fc19915" containerName="payload-pre-processing"
Oct 30 12:46:40.531400 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.531161    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c31a64e-d5e2-4883-a3b2-73631dbf5254" containerName="maas-api"
Oct 30 12:46:40.536272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.536253    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.557394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.557371    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"]
Oct 30 12:46:40.566046 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.566027    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c81f42-openshift-default-dockercfg-7fkzs\""
Oct 30 12:46:40.583467 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.583440    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:40.613346 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.613320    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.613466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.613374    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.613466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.613453    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcgmf\" (UniqueName: \"kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-kube-api-access-wcgmf\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.613541 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.613519    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.613593 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.613579    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.613628 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.613602    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.613682 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.613670    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.613714 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.613693    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.613745 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.613722    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.702130 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.702098    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:40.714798 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.714767    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.714919 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.714803    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.714919 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.714835    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.714919 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.714872    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.714919 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.714900    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.715125 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.714972    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcgmf\" (UniqueName: \"kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-kube-api-access-wcgmf\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.715125 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.715007    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.715125 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.715078    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.715125 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.715109    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.715327 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.715309    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.715327 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.715321    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.715532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.715512    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.715672 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.715654    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.715763 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.715746    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.717655 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.717635    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.717858 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.717838    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.744999 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.744931    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcgmf\" (UniqueName: \"kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-kube-api-access-wcgmf\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.747989 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.747950    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.849902 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.849808    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:40.862201 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:40.862161    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq" podUID="92a9d049-20c0-42fa-978b-169ed1c58b53" containerName="payload-processing" containerID="cri-o://3c05548f68281fa3475dbbc9f67583d03b2adb716508ec5b011770984c9b1f43" gracePeriod=25
Oct 30 12:46:41.026398 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.025007    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"]
Oct 30 12:46:41.029395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.028770    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.049744 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.049695    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-105228-openshift-default-dockercfg-kvxgn\""
Oct 30 12:46:41.060814 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.060787    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"]
Oct 30 12:46:41.073879 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.073853    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:41.114263 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.114240    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"
Oct 30 12:46:41.119387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.119348    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.119500 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.119451    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.119500 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.119488    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.119582 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.119513    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.119582 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.119548    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.119649 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.119606    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd7w7\" (UniqueName: \"kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-kube-api-access-sd7w7\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.119683 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.119645    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.119719 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.119694    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e76d3f0-c1f4-4458-be8d-8d6922001217-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.119751 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.119732    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.167541 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:41.167518    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode726d2a8_8f05_471d_a7d1_2fa2c8df6550.slice/crio-a1f5601aafb7deeee83f048e75f3359a13417e9a3fdb75f555520dab0f14ba76 WatchSource:0}: Error finding container a1f5601aafb7deeee83f048e75f3359a13417e9a3fdb75f555520dab0f14ba76: Status 404 returned error can't find the container with id a1f5601aafb7deeee83f048e75f3359a13417e9a3fdb75f555520dab0f14ba76
Oct 30 12:46:41.168240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.168217    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"]
Oct 30 12:46:41.169955 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.169918    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:41.170022 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.170003    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:41.170069 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.170053    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:41.180097 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.180075    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:41.222148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.220689    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a9d049-20c0-42fa-978b-169ed1c58b53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a9d049-20c0-42fa-978b-169ed1c58b53-plugins-config-volume\") pod \"92a9d049-20c0-42fa-978b-169ed1c58b53\" (UID: \"92a9d049-20c0-42fa-978b-169ed1c58b53\") "
Oct 30 12:46:41.222148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.220763    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a9d049-20c0-42fa-978b-169ed1c58b53-kube-api-access-zqqrn\" (UniqueName: \"kubernetes.io/projected/92a9d049-20c0-42fa-978b-169ed1c58b53-kube-api-access-zqqrn\") pod \"92a9d049-20c0-42fa-978b-169ed1c58b53\" (UID: \"92a9d049-20c0-42fa-978b-169ed1c58b53\") "
Oct 30 12:46:41.222148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.221060    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.222148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.221096    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.222148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.221132    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.222148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.221421    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.222148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.221510    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd7w7\" (UniqueName: \"kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-kube-api-access-sd7w7\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.222148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.221580    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.222148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.221635    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e76d3f0-c1f4-4458-be8d-8d6922001217-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.222148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.221669    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.222148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.221720    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.223234 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.223131    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.224474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.223516    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a9d049-20c0-42fa-978b-169ed1c58b53-plugins-config-volume" pod "92a9d049-20c0-42fa-978b-169ed1c58b53" (UID: "92a9d049-20c0-42fa-978b-169ed1c58b53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:41.224474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.223545    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.224474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.223804    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.224474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.224195    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e76d3f0-c1f4-4458-be8d-8d6922001217-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.224874 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.224838    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.255197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.253999    4522 generic.go:356] "Generic (PLEG): container finished" podID="92a9d049-20c0-42fa-978b-169ed1c58b53" containerID="3c05548f68281fa3475dbbc9f67583d03b2adb716508ec5b011770984c9b1f43" exitCode=0
Oct 30 12:46:41.255197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.254159    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq" event={"ID":"92a9d049-20c0-42fa-978b-169ed1c58b53","Type":"ContainerDied","Data":"3c05548f68281fa3475dbbc9f67583d03b2adb716508ec5b011770984c9b1f43"}
Oct 30 12:46:41.255197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.254215    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq" event={"ID":"92a9d049-20c0-42fa-978b-169ed1c58b53","Type":"ContainerDied","Data":"0ef87311254b5bcc37fb3949bf0af1592375aa650bc0d0e60a9cf07b5857e372"}
Oct 30 12:46:41.255197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.254248    4522 scope.go:122] "RemoveContainer" containerID="3c05548f68281fa3475dbbc9f67583d03b2adb716508ec5b011770984c9b1f43"
Oct 30 12:46:41.255197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.254422    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"
Oct 30 12:46:41.260377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.260006    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" event={"ID":"e726d2a8-8f05-471d-a7d1-2fa2c8df6550","Type":"ContainerStarted","Data":"a1f5601aafb7deeee83f048e75f3359a13417e9a3fdb75f555520dab0f14ba76"}
Oct 30 12:46:41.262731 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.262688    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.263613 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.263585    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.263987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.263847    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.263987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.263956    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a9d049-20c0-42fa-978b-169ed1c58b53-kube-api-access-zqqrn" pod "92a9d049-20c0-42fa-978b-169ed1c58b53" (UID: "92a9d049-20c0-42fa-978b-169ed1c58b53"). InnerVolumeSpecName "kube-api-access-zqqrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:41.266167 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.266142    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd7w7\" (UniqueName: \"kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-kube-api-access-sd7w7\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.276372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.275511    4522 scope.go:122] "RemoveContainer" containerID="3c05548f68281fa3475dbbc9f67583d03b2adb716508ec5b011770984c9b1f43"
Oct 30 12:46:41.276563 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:41.276534    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c05548f68281fa3475dbbc9f67583d03b2adb716508ec5b011770984c9b1f43\": container with ID starting with 3c05548f68281fa3475dbbc9f67583d03b2adb716508ec5b011770984c9b1f43 not found: ID does not exist" containerID="3c05548f68281fa3475dbbc9f67583d03b2adb716508ec5b011770984c9b1f43"
Oct 30 12:46:41.276674 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.276564    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c05548f68281fa3475dbbc9f67583d03b2adb716508ec5b011770984c9b1f43"} err="failed to get container status \"3c05548f68281fa3475dbbc9f67583d03b2adb716508ec5b011770984c9b1f43\": rpc error: code = NotFound desc = could not find container \"3c05548f68281fa3475dbbc9f67583d03b2adb716508ec5b011770984c9b1f43\": container with ID starting with 3c05548f68281fa3475dbbc9f67583d03b2adb716508ec5b011770984c9b1f43 not found: ID does not exist"
Oct 30 12:46:41.323243 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.323214    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a9d049-20c0-42fa-978b-169ed1c58b53-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:41.323243 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.323238    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqqrn\" (UniqueName: \"kubernetes.io/projected/92a9d049-20c0-42fa-978b-169ed1c58b53-kube-api-access-zqqrn\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:41.345541 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.345511    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:41.537712 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.537677    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:41.599228 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.599197    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"]
Oct 30 12:46:41.606616 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.606590    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4da58cb4-5bf47cf84-d8gbq"]
Oct 30 12:46:41.656445 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.656173    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:41.670374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.665751    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"]
Oct 30 12:46:41.677387 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:41.677345    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e76d3f0_c1f4_4458_be8d_8d6922001217.slice/crio-f03bce4d9878e9772b485387878b85e9eed8664eaf832672f1b10ddaee662773 WatchSource:0}: Error finding container f03bce4d9878e9772b485387878b85e9eed8664eaf832672f1b10ddaee662773: Status 404 returned error can't find the container with id f03bce4d9878e9772b485387878b85e9eed8664eaf832672f1b10ddaee662773
Oct 30 12:46:41.679920 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.679883    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:41.680014 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.679952    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:41.680014 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.679983    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:41.699727 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.699633    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"]
Oct 30 12:46:41.699870 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.699846    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w" podUID="a45d37fb-c68f-4347-b922-4096609dab99" containerName="maas-api" containerID="cri-o://b8f77a5ae892cbe5a61b1cdeec040c9d7b68528ee2adc2ddd952a211b17cc019" gracePeriod=30
Oct 30 12:46:41.777678 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.777648    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:41.954133 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:41.954111    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:42.030961 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.030929    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-maas-api-tls\") pod \"a45d37fb-c68f-4347-b922-4096609dab99\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") "
Oct 30 12:46:42.031122 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.031016    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a45d37fb-c68f-4347-b922-4096609dab99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a45d37fb-c68f-4347-b922-4096609dab99-openshift-service-ca\") pod \"a45d37fb-c68f-4347-b922-4096609dab99\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") "
Oct 30 12:46:42.031122 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.031073    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a45d37fb-c68f-4347-b922-4096609dab99-kube-api-access-kjkcl\" (UniqueName: \"kubernetes.io/projected/a45d37fb-c68f-4347-b922-4096609dab99-kube-api-access-kjkcl\") pod \"a45d37fb-c68f-4347-b922-4096609dab99\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") "
Oct 30 12:46:42.031211 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.031139    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-metrics-certs\") pod \"a45d37fb-c68f-4347-b922-4096609dab99\" (UID: \"a45d37fb-c68f-4347-b922-4096609dab99\") "
Oct 30 12:46:42.031438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.031408    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45d37fb-c68f-4347-b922-4096609dab99-openshift-service-ca" pod "a45d37fb-c68f-4347-b922-4096609dab99" (UID: "a45d37fb-c68f-4347-b922-4096609dab99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:42.031594 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.031577    4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a45d37fb-c68f-4347-b922-4096609dab99-openshift-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:42.033120 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.033094    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-maas-api-tls" pod "a45d37fb-c68f-4347-b922-4096609dab99" (UID: "a45d37fb-c68f-4347-b922-4096609dab99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:42.033583 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.033564    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-metrics-certs" pod "a45d37fb-c68f-4347-b922-4096609dab99" (UID: "a45d37fb-c68f-4347-b922-4096609dab99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:42.033671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.033658    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45d37fb-c68f-4347-b922-4096609dab99-kube-api-access-kjkcl" pod "a45d37fb-c68f-4347-b922-4096609dab99" (UID: "a45d37fb-c68f-4347-b922-4096609dab99"). InnerVolumeSpecName "kube-api-access-kjkcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:42.140379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.138002    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-maas-api-tls\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:42.140379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.138043    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjkcl\" (UniqueName: \"kubernetes.io/projected/a45d37fb-c68f-4347-b922-4096609dab99-kube-api-access-kjkcl\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:42.140379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.138058    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a45d37fb-c68f-4347-b922-4096609dab99-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:42.269417 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.269382    4522 generic.go:356] "Generic (PLEG): container finished" podID="a45d37fb-c68f-4347-b922-4096609dab99" containerID="b8f77a5ae892cbe5a61b1cdeec040c9d7b68528ee2adc2ddd952a211b17cc019" exitCode=0
Oct 30 12:46:42.269596 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.269427    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"
Oct 30 12:46:42.269596 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.269446    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w" event={"ID":"a45d37fb-c68f-4347-b922-4096609dab99","Type":"ContainerDied","Data":"b8f77a5ae892cbe5a61b1cdeec040c9d7b68528ee2adc2ddd952a211b17cc019"}
Oct 30 12:46:42.269596 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.269534    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w" event={"ID":"a45d37fb-c68f-4347-b922-4096609dab99","Type":"ContainerDied","Data":"a673cf117912c95680932cfb8bd2d81ca82e1be5692e1db23b21dbd93ff8f1da"}
Oct 30 12:46:42.269596 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.269568    4522 scope.go:122] "RemoveContainer" containerID="b8f77a5ae892cbe5a61b1cdeec040c9d7b68528ee2adc2ddd952a211b17cc019"
Oct 30 12:46:42.271233 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.271201    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" event={"ID":"e726d2a8-8f05-471d-a7d1-2fa2c8df6550","Type":"ContainerStarted","Data":"21dac7f2de25108ea47a5b9d35144a09b50a83a933a16cbaebbaf5fdf305bb2a"}
Oct 30 12:46:42.272776 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.272751    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" event={"ID":"8e76d3f0-c1f4-4458-be8d-8d6922001217","Type":"ContainerStarted","Data":"d61531e53dc65dbe5434cb28f9413bd502ceff2b7e1cd96f0a8e2a9166657fd6"}
Oct 30 12:46:42.272862 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.272801    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" event={"ID":"8e76d3f0-c1f4-4458-be8d-8d6922001217","Type":"ContainerStarted","Data":"f03bce4d9878e9772b485387878b85e9eed8664eaf832672f1b10ddaee662773"}
Oct 30 12:46:42.278981 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.278964    4522 scope.go:122] "RemoveContainer" containerID="b8f77a5ae892cbe5a61b1cdeec040c9d7b68528ee2adc2ddd952a211b17cc019"
Oct 30 12:46:42.279238 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:42.279221    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f77a5ae892cbe5a61b1cdeec040c9d7b68528ee2adc2ddd952a211b17cc019\": container with ID starting with b8f77a5ae892cbe5a61b1cdeec040c9d7b68528ee2adc2ddd952a211b17cc019 not found: ID does not exist" containerID="b8f77a5ae892cbe5a61b1cdeec040c9d7b68528ee2adc2ddd952a211b17cc019"
Oct 30 12:46:42.279284 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.279245    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f77a5ae892cbe5a61b1cdeec040c9d7b68528ee2adc2ddd952a211b17cc019"} err="failed to get container status \"b8f77a5ae892cbe5a61b1cdeec040c9d7b68528ee2adc2ddd952a211b17cc019\": rpc error: code = NotFound desc = could not find container \"b8f77a5ae892cbe5a61b1cdeec040c9d7b68528ee2adc2ddd952a211b17cc019\": container with ID starting with b8f77a5ae892cbe5a61b1cdeec040c9d7b68528ee2adc2ddd952a211b17cc019 not found: ID does not exist"
Oct 30 12:46:42.318535 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.318513    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"]
Oct 30 12:46:42.324248 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.324227    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1b2ed0a-869fc67fb-s495w"]
Oct 30 12:46:42.332296 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.332272    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a9d049-20c0-42fa-978b-169ed1c58b53" path="/var/lib/kubelet/pods/92a9d049-20c0-42fa-978b-169ed1c58b53/volumes"
Oct 30 12:46:42.332774 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.332755    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a45d37fb-c68f-4347-b922-4096609dab99" path="/var/lib/kubelet/pods/a45d37fb-c68f-4347-b922-4096609dab99/volumes"
Oct 30 12:46:42.346401 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.346345    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:42.351526 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.351509    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:42.363091 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.363051    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" podStartSLOduration=2.363039602 podStartE2EDuration="2.363039602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:42.361520529 +0000 UTC m=+646.176847018" watchObservedRunningTime="2026-10-30 12:46:42.363039602 +0000 UTC m=+646.178366091"
Oct 30 12:46:42.406319 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.406276    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" podStartSLOduration=2.406261465 podStartE2EDuration="2.406261465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:42.405448342 +0000 UTC m=+646.220774830" watchObservedRunningTime="2026-10-30 12:46:42.406261465 +0000 UTC m=+646.221587955"
Oct 30 12:46:42.445924 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.445890    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:42.794508 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.794468    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:42.851073 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.851045    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:42.856278 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.856257    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:42.908364 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:42.908332    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:43.278520 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:43.278491    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:43.278692 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:43.278550    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:43.279211 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:43.279182    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:46:43.279399 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:43.279381    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:46:43.491764 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:43.491731    4522 patch_prober.go:29] interesting pod/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 12:46:43.492102 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:43.491782    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:46:44.405500 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:44.405457    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:44.479100 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:44.479067    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"]
Oct 30 12:46:44.479324 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:44.479301    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" podUID="1f227556-8b10-40b3-aa88-cf825443e409" containerName="istio-proxy" containerID="cri-o://1acde284caca1c355f23bf973cd1506c3ad2fa211dac092de421fd942741a2fc" gracePeriod=30
Oct 30 12:46:44.505909 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:44.505867    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:44.761657 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:44.761577    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:44.867684 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:44.867645    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:45.027755 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.027734    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:45.179543 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.179514    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-data\") pod \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") "
Oct 30 12:46:45.179724 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.179582    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-credential-socket\") pod \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") "
Oct 30 12:46:45.179724 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.179625    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-token\" (UniqueName: \"kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-token\") pod \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") "
Oct 30 12:46:45.179724 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.179655    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-socket\") pod \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") "
Oct 30 12:46:45.179724 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.179686    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-podinfo\") pod \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") "
Oct 30 12:46:45.179724 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.179713    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-kube-api-access-9gkjk\" (UniqueName: \"kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-kube-api-access-9gkjk\") pod \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") "
Oct 30 12:46:45.179992 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.179737    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-envoy\") pod \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") "
Oct 30 12:46:45.179992 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.179774    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-certs\") pod \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") "
Oct 30 12:46:45.179992 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.179812    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istiod-ca-cert\") pod \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\" (UID: \"d31d98d4-f212-43b5-8af6-e40a792bdcdf\") "
Oct 30 12:46:45.180158 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.179768    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-data" pod "d31d98d4-f212-43b5-8af6-e40a792bdcdf" (UID: "d31d98d4-f212-43b5-8af6-e40a792bdcdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:45.180158 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.180061    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-certs" pod "d31d98d4-f212-43b5-8af6-e40a792bdcdf" (UID: "d31d98d4-f212-43b5-8af6-e40a792bdcdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:45.180158 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.179889    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-credential-socket" pod "d31d98d4-f212-43b5-8af6-e40a792bdcdf" (UID: "d31d98d4-f212-43b5-8af6-e40a792bdcdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:45.180158 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.180042    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-socket" pod "d31d98d4-f212-43b5-8af6-e40a792bdcdf" (UID: "d31d98d4-f212-43b5-8af6-e40a792bdcdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:45.180389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.180253    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:45.180389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.180272    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:45.180389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.180286    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:45.180389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.180299    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:45.180389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.180313    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istiod-ca-cert" pod "d31d98d4-f212-43b5-8af6-e40a792bdcdf" (UID: "d31d98d4-f212-43b5-8af6-e40a792bdcdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:45.181899 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.181870    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-kube-api-access-9gkjk" pod "d31d98d4-f212-43b5-8af6-e40a792bdcdf" (UID: "d31d98d4-f212-43b5-8af6-e40a792bdcdf"). InnerVolumeSpecName "kube-api-access-9gkjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:45.181899 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.181868    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-envoy" pod "d31d98d4-f212-43b5-8af6-e40a792bdcdf" (UID: "d31d98d4-f212-43b5-8af6-e40a792bdcdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:45.182107 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.182087    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-podinfo" pod "d31d98d4-f212-43b5-8af6-e40a792bdcdf" (UID: "d31d98d4-f212-43b5-8af6-e40a792bdcdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:46:45.182156 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.182124    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-token" pod "d31d98d4-f212-43b5-8af6-e40a792bdcdf" (UID: "d31d98d4-f212-43b5-8af6-e40a792bdcdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:45.281085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.281061    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:45.281085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.281086    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:45.281252 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.281095    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:45.281252 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.281102    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31d98d4-f212-43b5-8af6-e40a792bdcdf-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:45.281252 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.281111    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gkjk\" (UniqueName: \"kubernetes.io/projected/d31d98d4-f212-43b5-8af6-e40a792bdcdf-kube-api-access-9gkjk\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:45.288492 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.288467    4522 generic.go:356] "Generic (PLEG): container finished" podID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerID="2b76a50affa3e4ebfcfa072f51a73b0e34e9384be6a815b5ed56a671f171cf48" exitCode=0
Oct 30 12:46:45.288603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.288519    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"
Oct 30 12:46:45.288603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.288547    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" event={"ID":"d31d98d4-f212-43b5-8af6-e40a792bdcdf","Type":"ContainerDied","Data":"2b76a50affa3e4ebfcfa072f51a73b0e34e9384be6a815b5ed56a671f171cf48"}
Oct 30 12:46:45.288690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.288603    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4" event={"ID":"d31d98d4-f212-43b5-8af6-e40a792bdcdf","Type":"ContainerDied","Data":"1a8016b754a19dd24e6b33590e3ab1389a517df9c30c346c68f47ebe79e995ea"}
Oct 30 12:46:45.288690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.288628    4522 scope.go:122] "RemoveContainer" containerID="2b76a50affa3e4ebfcfa072f51a73b0e34e9384be6a815b5ed56a671f171cf48"
Oct 30 12:46:45.296663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.296422    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:45.298086 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.298069    4522 scope.go:122] "RemoveContainer" containerID="7cc68a547ac657bee3768687515a54d8ae444ee3bab506bd2f25bc4a7a9453ad"
Oct 30 12:46:45.305492 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.305476    4522 scope.go:122] "RemoveContainer" containerID="2b76a50affa3e4ebfcfa072f51a73b0e34e9384be6a815b5ed56a671f171cf48"
Oct 30 12:46:45.305747 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:45.305729    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b76a50affa3e4ebfcfa072f51a73b0e34e9384be6a815b5ed56a671f171cf48\": container with ID starting with 2b76a50affa3e4ebfcfa072f51a73b0e34e9384be6a815b5ed56a671f171cf48 not found: ID does not exist" containerID="2b76a50affa3e4ebfcfa072f51a73b0e34e9384be6a815b5ed56a671f171cf48"
Oct 30 12:46:45.305789 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.305757    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b76a50affa3e4ebfcfa072f51a73b0e34e9384be6a815b5ed56a671f171cf48"} err="failed to get container status \"2b76a50affa3e4ebfcfa072f51a73b0e34e9384be6a815b5ed56a671f171cf48\": rpc error: code = NotFound desc = could not find container \"2b76a50affa3e4ebfcfa072f51a73b0e34e9384be6a815b5ed56a671f171cf48\": container with ID starting with 2b76a50affa3e4ebfcfa072f51a73b0e34e9384be6a815b5ed56a671f171cf48 not found: ID does not exist"
Oct 30 12:46:45.305789 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.305773    4522 scope.go:122] "RemoveContainer" containerID="7cc68a547ac657bee3768687515a54d8ae444ee3bab506bd2f25bc4a7a9453ad"
Oct 30 12:46:45.305976 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:45.305960    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc68a547ac657bee3768687515a54d8ae444ee3bab506bd2f25bc4a7a9453ad\": container with ID starting with 7cc68a547ac657bee3768687515a54d8ae444ee3bab506bd2f25bc4a7a9453ad not found: ID does not exist" containerID="7cc68a547ac657bee3768687515a54d8ae444ee3bab506bd2f25bc4a7a9453ad"
Oct 30 12:46:45.306116 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.305981    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc68a547ac657bee3768687515a54d8ae444ee3bab506bd2f25bc4a7a9453ad"} err="failed to get container status \"7cc68a547ac657bee3768687515a54d8ae444ee3bab506bd2f25bc4a7a9453ad\": rpc error: code = NotFound desc = could not find container \"7cc68a547ac657bee3768687515a54d8ae444ee3bab506bd2f25bc4a7a9453ad\": container with ID starting with 7cc68a547ac657bee3768687515a54d8ae444ee3bab506bd2f25bc4a7a9453ad not found: ID does not exist"
Oct 30 12:46:45.337396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.337368    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"]
Oct 30 12:46:45.352133 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.351710    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4da58cb4-openshift-default-64bc999667-tbvq4"]
Oct 30 12:46:45.701998 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.701926    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"]
Oct 30 12:46:45.702372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.702332    4522 cpu_manager.go:414] "containerMap: removing container" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy"
Oct 30 12:46:45.702372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.702343    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy"
Oct 30 12:46:45.702372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.702371    4522 cpu_manager.go:414] "containerMap: removing container" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy"
Oct 30 12:46:45.702499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.702378    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy"
Oct 30 12:46:45.702499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.702394    4522 cpu_manager.go:414] "containerMap: removing container" podUID="92a9d049-20c0-42fa-978b-169ed1c58b53" containerName="payload-processing"
Oct 30 12:46:45.702499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.702401    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a9d049-20c0-42fa-978b-169ed1c58b53" containerName="payload-processing"
Oct 30 12:46:45.702499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.702420    4522 cpu_manager.go:414] "containerMap: removing container" podUID="a45d37fb-c68f-4347-b922-4096609dab99" containerName="maas-api"
Oct 30 12:46:45.702499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.702425    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a45d37fb-c68f-4347-b922-4096609dab99" containerName="maas-api"
Oct 30 12:46:45.702499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.702480    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy"
Oct 30 12:46:45.702499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.702487    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" containerName="istio-proxy"
Oct 30 12:46:45.702499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.702493    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a9d049-20c0-42fa-978b-169ed1c58b53" containerName="payload-processing"
Oct 30 12:46:45.702499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.702501    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a45d37fb-c68f-4347-b922-4096609dab99" containerName="maas-api"
Oct 30 12:46:45.707131 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.707111    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.721604 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.721580    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-53dcb1-openshift-default-dockercfg-nlgks\""
Oct 30 12:46:45.735269 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.735244    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"]
Oct 30 12:46:45.759142 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.759109    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:45.785595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.785569    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.785743 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.785609    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.785743 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.785638    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.785743 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.785711    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.785874 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.785753    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.785874 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.785836    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.785874 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.785861    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.785991 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.785877    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcffs\" (UniqueName: \"kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-kube-api-access-jcffs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.785991 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.785928    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.871113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.871077    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:45.887391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.887334    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.887532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.887427    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.887532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.887459    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.887532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.887506    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.887710 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.887534    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.887710 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.887564    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.887710 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.887589    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcffs\" (UniqueName: \"kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-kube-api-access-jcffs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.887710 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.887643    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.888249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.887750    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.888249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.887863    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.888249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.887981    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.888249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.888046    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.888249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.888203    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.888249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.888222    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.890076 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.890029    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.890520 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.890495    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.928199 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.928158    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcffs\" (UniqueName: \"kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-kube-api-access-jcffs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:45.949202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:45.949168    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:46.022404 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.022342    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:46.334132 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.334041    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31d98d4-f212-43b5-8af6-e40a792bdcdf" path="/var/lib/kubelet/pods/d31d98d4-f212-43b5-8af6-e40a792bdcdf/volumes"
Oct 30 12:46:46.335816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.335793    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"]
Oct 30 12:46:46.337188 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:46.337160    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e41433e_dd8f_4a3b_aaa2_9f3c8aac3bcc.slice/crio-763451398dcad89c4187748f47fb92a68d4ac0c199a86e93a13db3ef02e604a7 WatchSource:0}: Error finding container 763451398dcad89c4187748f47fb92a68d4ac0c199a86e93a13db3ef02e604a7: Status 404 returned error can't find the container with id 763451398dcad89c4187748f47fb92a68d4ac0c199a86e93a13db3ef02e604a7
Oct 30 12:46:46.339244 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.339211    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:46.339313 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.339271    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:46.339313 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.339299    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:46.453243 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.453213    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:46.555307 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.555267    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:46.766410 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.766311    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"]
Oct 30 12:46:46.799709 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.799675    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"]
Oct 30 12:46:46.800062 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.799973    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"
Oct 30 12:46:46.815232 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.815203    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-105228\""
Oct 30 12:46:46.827059 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.827032    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-105228-dockercfg-sjn4l\""
Oct 30 12:46:46.900537 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.900500    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g495\" (UniqueName: \"kubernetes.io/projected/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-kube-api-access-8g495\") pod \"payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf\" (UID: \"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"
Oct 30 12:46:46.900686 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:46.900625    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf\" (UID: \"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"
Oct 30 12:46:47.001220 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:47.001181    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g495\" (UniqueName: \"kubernetes.io/projected/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-kube-api-access-8g495\") pod \"payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf\" (UID: \"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"
Oct 30 12:46:47.001445 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:47.001258    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf\" (UID: \"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"
Oct 30 12:46:47.002003 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:47.001969    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf\" (UID: \"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"
Oct 30 12:46:47.006089 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:47.006061    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:47.020297 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:47.020273    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g495\" (UniqueName: \"kubernetes.io/projected/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-kube-api-access-8g495\") pod \"payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf\" (UID: \"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"
Oct 30 12:46:47.118585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:47.118547    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"
Oct 30 12:46:47.286119 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:47.286019    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:46:47.301399 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:47.301369    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" event={"ID":"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc","Type":"ContainerStarted","Data":"3f884ee20ba3ffb2b95a8d49b5d9776fe6132a9bc34156ab2377b68194825847"}
Oct 30 12:46:47.301550 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:47.301417    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" event={"ID":"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc","Type":"ContainerStarted","Data":"763451398dcad89c4187748f47fb92a68d4ac0c199a86e93a13db3ef02e604a7"}
Oct 30 12:46:47.348197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:47.348105    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" podStartSLOduration=2.348085896 podStartE2EDuration="2.348085896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:47.346252552 +0000 UTC m=+651.161579041" watchObservedRunningTime="2026-10-30 12:46:47.348085896 +0000 UTC m=+651.163412384"
Oct 30 12:46:47.432114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:47.432086    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"]
Oct 30 12:46:47.435014 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:47.434980    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb530d6b3_7fb9_4e37_ba74_c9a1bb0d3878.slice/crio-4b00298e2b4b135a19064805bf51d778c6c7ad35dd0a5219a330f33fc02c96a8 WatchSource:0}: Error finding container 4b00298e2b4b135a19064805bf51d778c6c7ad35dd0a5219a330f33fc02c96a8: Status 404 returned error can't find the container with id 4b00298e2b4b135a19064805bf51d778c6c7ad35dd0a5219a330f33fc02c96a8
Oct 30 12:46:48.028421 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.024776    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:48.034061 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.034032    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:48.307602 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.307510    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf" event={"ID":"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878","Type":"ContainerStarted","Data":"4013df4a250d1bfaafba6e11f197c44b137b189749337352107e8389d1f5f42c"}
Oct 30 12:46:48.307602 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.307565    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf" event={"ID":"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878","Type":"ContainerStarted","Data":"4b00298e2b4b135a19064805bf51d778c6c7ad35dd0a5219a330f33fc02c96a8"}
Oct 30 12:46:48.307864 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.307840    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"
Oct 30 12:46:48.307909 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.307893    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:48.308864 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.308843    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:46:48.345777 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.345734    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf" podStartSLOduration=2.345721437 podStartE2EDuration="2.345721437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:48.343977447 +0000 UTC m=+652.159303936" watchObservedRunningTime="2026-10-30 12:46:48.345721437 +0000 UTC m=+652.161047926"
Oct 30 12:46:48.402917 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.402660    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"]
Oct 30 12:46:48.406989 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.406960    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:48.428594 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.428565    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"]
Oct 30 12:46:48.428713 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.428681    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-105228-serving-cert\""
Oct 30 12:46:48.428800 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.428786    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mwj6w\""
Oct 30 12:46:48.428864 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.428852    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:46:48.428942 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.428931    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:46:48.429047 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.429035    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 12:46:48.485737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.485670    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:48.521465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.521439    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-metrics-certs\") pod \"maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:48.521590 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.521481    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-maas-api-tls\") pod \"maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:48.521648 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.521611    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/753b333f-cc81-4df9-ad51-e14445ce8fa4-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:48.521648 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.521638    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbl5q\" (UniqueName: \"kubernetes.io/projected/753b333f-cc81-4df9-ad51-e14445ce8fa4-kube-api-access-sbl5q\") pod \"maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:48.583253 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.583170    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:48.622483 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.622457    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/753b333f-cc81-4df9-ad51-e14445ce8fa4-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:48.622604 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.622491    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbl5q\" (UniqueName: \"kubernetes.io/projected/753b333f-cc81-4df9-ad51-e14445ce8fa4-kube-api-access-sbl5q\") pod \"maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:48.622604 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.622545    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-metrics-certs\") pod \"maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:48.622604 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.622572    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-maas-api-tls\") pod \"maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:48.623176 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.623151    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/753b333f-cc81-4df9-ad51-e14445ce8fa4-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:48.624899 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.624882    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-maas-api-tls\") pod \"maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:48.625046 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.625026    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-metrics-certs\") pod \"maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:48.640935 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.640911    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbl5q\" (UniqueName: \"kubernetes.io/projected/753b333f-cc81-4df9-ad51-e14445ce8fa4-kube-api-access-sbl5q\") pod \"maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:48.695381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.687861    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:48.723565 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:48.723530    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:49.019011 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.018984    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"]
Oct 30 12:46:49.019294 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:49.019271    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753b333f_cc81_4df9_ad51_e14445ce8fa4.slice/crio-bbf9f1a89836a97aa010eb2a7a823b2ef78161d634b585f22707a833db34fb97 WatchSource:0}: Error finding container bbf9f1a89836a97aa010eb2a7a823b2ef78161d634b585f22707a833db34fb97: Status 404 returned error can't find the container with id bbf9f1a89836a97aa010eb2a7a823b2ef78161d634b585f22707a833db34fb97
Oct 30 12:46:49.192693 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.192662    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"]
Oct 30 12:46:49.292157 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.292128    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:49.313529 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.313497    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5" event={"ID":"753b333f-cc81-4df9-ad51-e14445ce8fa4","Type":"ContainerStarted","Data":"bbf9f1a89836a97aa010eb2a7a823b2ef78161d634b585f22707a833db34fb97"}
Oct 30 12:46:49.411919 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.405182    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:49.782621 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.782565    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:49.821977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.821952    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:49.896151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.896079    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:49.957223 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.957193    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-socket\") pod \"1f227556-8b10-40b3-aa88-cf825443e409\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") "
Oct 30 12:46:49.957381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.957233    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-certs\") pod \"1f227556-8b10-40b3-aa88-cf825443e409\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") "
Oct 30 12:46:49.957381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.957260    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-kube-api-access-bt6rf\" (UniqueName: \"kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-kube-api-access-bt6rf\") pod \"1f227556-8b10-40b3-aa88-cf825443e409\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") "
Oct 30 12:46:49.957381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.957314    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f227556-8b10-40b3-aa88-cf825443e409-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f227556-8b10-40b3-aa88-cf825443e409-istiod-ca-cert\") pod \"1f227556-8b10-40b3-aa88-cf825443e409\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") "
Oct 30 12:46:49.957531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.957457    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f227556-8b10-40b3-aa88-cf825443e409-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f227556-8b10-40b3-aa88-cf825443e409-istio-podinfo\") pod \"1f227556-8b10-40b3-aa88-cf825443e409\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") "
Oct 30 12:46:49.957531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.957489    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-socket" pod "1f227556-8b10-40b3-aa88-cf825443e409" (UID: "1f227556-8b10-40b3-aa88-cf825443e409"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:49.957531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.957523    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-envoy\") pod \"1f227556-8b10-40b3-aa88-cf825443e409\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") "
Oct 30 12:46:49.957643 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.957570    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-certs" pod "1f227556-8b10-40b3-aa88-cf825443e409" (UID: "1f227556-8b10-40b3-aa88-cf825443e409"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:49.957643 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.957588    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-data\") pod \"1f227556-8b10-40b3-aa88-cf825443e409\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") "
Oct 30 12:46:49.957740 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.957653    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-istio-token\" (UniqueName: \"kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-istio-token\") pod \"1f227556-8b10-40b3-aa88-cf825443e409\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") "
Oct 30 12:46:49.957740 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.957701    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-credential-socket\") pod \"1f227556-8b10-40b3-aa88-cf825443e409\" (UID: \"1f227556-8b10-40b3-aa88-cf825443e409\") "
Oct 30 12:46:49.957912 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.957788    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f227556-8b10-40b3-aa88-cf825443e409-istiod-ca-cert" pod "1f227556-8b10-40b3-aa88-cf825443e409" (UID: "1f227556-8b10-40b3-aa88-cf825443e409"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:49.957912 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.957878    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-data" pod "1f227556-8b10-40b3-aa88-cf825443e409" (UID: "1f227556-8b10-40b3-aa88-cf825443e409"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:49.958110 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.958085    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f227556-8b10-40b3-aa88-cf825443e409-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:49.958110 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.958108    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:49.958234 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.958121    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:49.958234 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.958135    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:49.958234 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.958083    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-credential-socket" pod "1f227556-8b10-40b3-aa88-cf825443e409" (UID: "1f227556-8b10-40b3-aa88-cf825443e409"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:49.959637 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.959610    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-kube-api-access-bt6rf" pod "1f227556-8b10-40b3-aa88-cf825443e409" (UID: "1f227556-8b10-40b3-aa88-cf825443e409"). InnerVolumeSpecName "kube-api-access-bt6rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:49.959637 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.959624    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f227556-8b10-40b3-aa88-cf825443e409-istio-podinfo" pod "1f227556-8b10-40b3-aa88-cf825443e409" (UID: "1f227556-8b10-40b3-aa88-cf825443e409"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:46:49.959929 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.959914    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-envoy" pod "1f227556-8b10-40b3-aa88-cf825443e409" (UID: "1f227556-8b10-40b3-aa88-cf825443e409"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:49.959967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:49.959928    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-istio-token" pod "1f227556-8b10-40b3-aa88-cf825443e409" (UID: "1f227556-8b10-40b3-aa88-cf825443e409"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:50.059391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.059346    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f227556-8b10-40b3-aa88-cf825443e409-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:50.059391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.059388    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:50.059519 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.059397    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:50.059519 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.059405    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f227556-8b10-40b3-aa88-cf825443e409-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:50.059519 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.059415    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt6rf\" (UniqueName: \"kubernetes.io/projected/1f227556-8b10-40b3-aa88-cf825443e409-kube-api-access-bt6rf\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:50.212785 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.212693    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:50.319660 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.319626    4522 generic.go:356] "Generic (PLEG): container finished" podID="1f227556-8b10-40b3-aa88-cf825443e409" containerID="1acde284caca1c355f23bf973cd1506c3ad2fa211dac092de421fd942741a2fc" exitCode=0
Oct 30 12:46:50.319804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.319673    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"
Oct 30 12:46:50.319804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.319698    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" event={"ID":"1f227556-8b10-40b3-aa88-cf825443e409","Type":"ContainerDied","Data":"1acde284caca1c355f23bf973cd1506c3ad2fa211dac092de421fd942741a2fc"}
Oct 30 12:46:50.319804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.319740    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv" event={"ID":"1f227556-8b10-40b3-aa88-cf825443e409","Type":"ContainerDied","Data":"1984963ca1b43373631c6b6e6e0213ba1076841cc60ff3ec1514eec30811c27c"}
Oct 30 12:46:50.319804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.319769    4522 scope.go:122] "RemoveContainer" containerID="1acde284caca1c355f23bf973cd1506c3ad2fa211dac092de421fd942741a2fc"
Oct 30 12:46:50.321378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.321343    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5" event={"ID":"753b333f-cc81-4df9-ad51-e14445ce8fa4","Type":"ContainerStarted","Data":"246e8f45056bee3ac666d1fdfcf150746af43f9f22bed74e1a8386554b349541"}
Oct 30 12:46:50.321494 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.321397    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:50.329255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.329237    4522 scope.go:122] "RemoveContainer" containerID="8cb0aaa0cf02bf785ec204aa7fa55002bdb32299b8b509464fb5c43f2f8ba69b"
Oct 30 12:46:50.337705 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.337689    4522 scope.go:122] "RemoveContainer" containerID="1acde284caca1c355f23bf973cd1506c3ad2fa211dac092de421fd942741a2fc"
Oct 30 12:46:50.337939 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:50.337921    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1acde284caca1c355f23bf973cd1506c3ad2fa211dac092de421fd942741a2fc\": container with ID starting with 1acde284caca1c355f23bf973cd1506c3ad2fa211dac092de421fd942741a2fc not found: ID does not exist" containerID="1acde284caca1c355f23bf973cd1506c3ad2fa211dac092de421fd942741a2fc"
Oct 30 12:46:50.338001 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.337945    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1acde284caca1c355f23bf973cd1506c3ad2fa211dac092de421fd942741a2fc"} err="failed to get container status \"1acde284caca1c355f23bf973cd1506c3ad2fa211dac092de421fd942741a2fc\": rpc error: code = NotFound desc = could not find container \"1acde284caca1c355f23bf973cd1506c3ad2fa211dac092de421fd942741a2fc\": container with ID starting with 1acde284caca1c355f23bf973cd1506c3ad2fa211dac092de421fd942741a2fc not found: ID does not exist"
Oct 30 12:46:50.338001 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.337963    4522 scope.go:122] "RemoveContainer" containerID="8cb0aaa0cf02bf785ec204aa7fa55002bdb32299b8b509464fb5c43f2f8ba69b"
Oct 30 12:46:50.338213 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:50.338190    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb0aaa0cf02bf785ec204aa7fa55002bdb32299b8b509464fb5c43f2f8ba69b\": container with ID starting with 8cb0aaa0cf02bf785ec204aa7fa55002bdb32299b8b509464fb5c43f2f8ba69b not found: ID does not exist" containerID="8cb0aaa0cf02bf785ec204aa7fa55002bdb32299b8b509464fb5c43f2f8ba69b"
Oct 30 12:46:50.338308 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.338217    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb0aaa0cf02bf785ec204aa7fa55002bdb32299b8b509464fb5c43f2f8ba69b"} err="failed to get container status \"8cb0aaa0cf02bf785ec204aa7fa55002bdb32299b8b509464fb5c43f2f8ba69b\": rpc error: code = NotFound desc = could not find container \"8cb0aaa0cf02bf785ec204aa7fa55002bdb32299b8b509464fb5c43f2f8ba69b\": container with ID starting with 8cb0aaa0cf02bf785ec204aa7fa55002bdb32299b8b509464fb5c43f2f8ba69b not found: ID does not exist"
Oct 30 12:46:50.371261 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.371235    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"]
Oct 30 12:46:50.383044 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.383018    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c1b2ed0a-openshift-default-76d4b649c-pbtcv"]
Oct 30 12:46:50.412536 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.412490    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5" podStartSLOduration=1.9882485669999999 podStartE2EDuration="2.412477799s" totalImagesPullingTime="424.229232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:50.410886131 +0000 UTC m=+654.226212629" watchObservedRunningTime="2026-10-30 12:46:50.412477799 +0000 UTC m=+654.227804287"
Oct 30 12:46:50.742502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.742474    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:50.854376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.853954    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:50.949548 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:50.949511    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:51.094371 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.094333    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:51.134534 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:46:51.134498    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:46:51.134898 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.134878    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:46:51.524466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.524420    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:51.635066 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.635033    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:51.761470 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.761428    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:51.884647 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.884509    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:51.969485 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.969454    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"]
Oct 30 12:46:51.969873 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.969861    4522 cpu_manager.go:414] "containerMap: removing container" podUID="1f227556-8b10-40b3-aa88-cf825443e409" containerName="istio-proxy"
Oct 30 12:46:51.969927 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.969875    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f227556-8b10-40b3-aa88-cf825443e409" containerName="istio-proxy"
Oct 30 12:46:51.969927 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.969896    4522 cpu_manager.go:414] "containerMap: removing container" podUID="1f227556-8b10-40b3-aa88-cf825443e409" containerName="istio-proxy"
Oct 30 12:46:51.969927 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.969901    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f227556-8b10-40b3-aa88-cf825443e409" containerName="istio-proxy"
Oct 30 12:46:51.970076 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.969982    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f227556-8b10-40b3-aa88-cf825443e409" containerName="istio-proxy"
Oct 30 12:46:51.974466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.974444    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:51.993292 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:51.993269    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4cd3fb-openshift-default-dockercfg-sd289\""
Oct 30 12:46:52.021860 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.021832    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"]
Oct 30 12:46:52.075944 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.075905    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:52.120225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.100936    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.120225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.100997    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfrcr\" (UniqueName: \"kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-kube-api-access-vfrcr\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.120225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.101049    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.120225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.101133    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674b4304-a144-449c-9c7b-bd026b669f74-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.120225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.101178    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.120225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.101221    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.120225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.101251    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674b4304-a144-449c-9c7b-bd026b669f74-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.120225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.101321    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.120225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.101366    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.202232 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.202147    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.202232 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.202192    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.202232 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.202231    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.202577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.202267    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfrcr\" (UniqueName: \"kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-kube-api-access-vfrcr\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.202577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.202333    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.202577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.202429    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674b4304-a144-449c-9c7b-bd026b669f74-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.202577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.202472    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.202577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.202518    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.202577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.202549    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674b4304-a144-449c-9c7b-bd026b669f74-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.202577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.202550    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.202577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.202573    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.202958 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.202719    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.203117 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.203097    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.203459 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.203440    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674b4304-a144-449c-9c7b-bd026b669f74-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.204794 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.204776    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.205114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.205095    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674b4304-a144-449c-9c7b-bd026b669f74-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.214642 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.214614    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:52.227847 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.227820    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.228033 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.228004    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfrcr\" (UniqueName: \"kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-kube-api-access-vfrcr\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.292330 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.292293    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:52.335061 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.335020    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f227556-8b10-40b3-aa88-cf825443e409" path="/var/lib/kubelet/pods/1f227556-8b10-40b3-aa88-cf825443e409/volumes"
Oct 30 12:46:52.414950 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.414913    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:52.596681 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.596645    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:52.618173 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.618141    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"]
Oct 30 12:46:52.619979 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:46:52.619950    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674b4304_a144_449c_9c7b_bd026b669f74.slice/crio-a38759b92adbc4489062845766c5778bcc45c8bbce6a7c24071d71a3a736341a WatchSource:0}: Error finding container a38759b92adbc4489062845766c5778bcc45c8bbce6a7c24071d71a3a736341a: Status 404 returned error can't find the container with id a38759b92adbc4489062845766c5778bcc45c8bbce6a7c24071d71a3a736341a
Oct 30 12:46:52.622177 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.622143    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:52.622242 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.622210    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:52.622293 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.622238    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:46:52.999902 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:52.999780    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:53.099983 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:53.099944    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:53.208131 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:53.205601    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:53.339877 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:53.339839    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" event={"ID":"674b4304-a144-449c-9c7b-bd026b669f74","Type":"ContainerStarted","Data":"f3c2d524d044dbc6913cdf84b7a740c31e43c2ccab9abef1f5deefd076ef2cef"}
Oct 30 12:46:53.340050 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:53.339894    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" event={"ID":"674b4304-a144-449c-9c7b-bd026b669f74","Type":"ContainerStarted","Data":"a38759b92adbc4489062845766c5778bcc45c8bbce6a7c24071d71a3a736341a"}
Oct 30 12:46:54.193570 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.193506    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s" podUID="9b7f9517-46e8-4254-bb3b-a6995326049a" containerName="payload-pre-processing" containerID="cri-o://4e55e36058cd20f0a91e4e11141884c7ed4d83af5c1888699999028562d80100" gracePeriod=25
Oct 30 12:46:54.292546 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.292516    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:54.298128 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.298106    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:54.347519 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.347487    4522 generic.go:356] "Generic (PLEG): container finished" podID="9b7f9517-46e8-4254-bb3b-a6995326049a" containerID="4e55e36058cd20f0a91e4e11141884c7ed4d83af5c1888699999028562d80100" exitCode=0
Oct 30 12:46:54.347651 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.347558    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s" event={"ID":"9b7f9517-46e8-4254-bb3b-a6995326049a","Type":"ContainerDied","Data":"4e55e36058cd20f0a91e4e11141884c7ed4d83af5c1888699999028562d80100"}
Oct 30 12:46:54.347770 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.347754    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:54.348645 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.348622    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:46:54.352534 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.352496    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" podStartSLOduration=3.352485741 podStartE2EDuration="3.352485741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:53.391237157 +0000 UTC m=+657.206563671" watchObservedRunningTime="2026-10-30 12:46:54.352485741 +0000 UTC m=+658.167812230"
Oct 30 12:46:54.540622 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.540585    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"
Oct 30 12:46:54.644942 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.644913    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b7f9517-46e8-4254-bb3b-a6995326049a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7f9517-46e8-4254-bb3b-a6995326049a-plugins-config-volume\") pod \"9b7f9517-46e8-4254-bb3b-a6995326049a\" (UID: \"9b7f9517-46e8-4254-bb3b-a6995326049a\") "
Oct 30 12:46:54.645120 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.645018    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7f9517-46e8-4254-bb3b-a6995326049a-kube-api-access-g7v7r\" (UniqueName: \"kubernetes.io/projected/9b7f9517-46e8-4254-bb3b-a6995326049a-kube-api-access-g7v7r\") pod \"9b7f9517-46e8-4254-bb3b-a6995326049a\" (UID: \"9b7f9517-46e8-4254-bb3b-a6995326049a\") "
Oct 30 12:46:54.645295 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.645272    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7f9517-46e8-4254-bb3b-a6995326049a-plugins-config-volume" pod "9b7f9517-46e8-4254-bb3b-a6995326049a" (UID: "9b7f9517-46e8-4254-bb3b-a6995326049a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:54.645447 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.645430    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7f9517-46e8-4254-bb3b-a6995326049a-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:54.647054 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.647035    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7f9517-46e8-4254-bb3b-a6995326049a-kube-api-access-g7v7r" pod "9b7f9517-46e8-4254-bb3b-a6995326049a" (UID: "9b7f9517-46e8-4254-bb3b-a6995326049a"). InnerVolumeSpecName "kube-api-access-g7v7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:54.746109 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.746039    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7v7r\" (UniqueName: \"kubernetes.io/projected/9b7f9517-46e8-4254-bb3b-a6995326049a-kube-api-access-g7v7r\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:46:54.915712 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:54.915679    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:55.097341 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:55.097253    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:55.223975 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:55.223944    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:55.353066 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:55.352995    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"
Oct 30 12:46:55.353066 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:55.353038    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s" event={"ID":"9b7f9517-46e8-4254-bb3b-a6995326049a","Type":"ContainerDied","Data":"33af60508310d397f55e381f69e90ad4a478d686d3c5a387372df51abbc1d5e4"}
Oct 30 12:46:55.353247 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:55.353094    4522 scope.go:122] "RemoveContainer" containerID="4e55e36058cd20f0a91e4e11141884c7ed4d83af5c1888699999028562d80100"
Oct 30 12:46:55.414182 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:55.414151    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"]
Oct 30 12:46:55.426156 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:55.426128    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-48623dd7-6f4c5fc555-7zs5s"]
Oct 30 12:46:55.801822 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:55.801784    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:55.928204 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:55.928169    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:56.440216 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:56.440184    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7f9517-46e8-4254-bb3b-a6995326049a" path="/var/lib/kubelet/pods/9b7f9517-46e8-4254-bb3b-a6995326049a/volumes"
Oct 30 12:46:56.440688 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:56.440630    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 12:46:56.813621 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:56.813591    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:56.955764 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:56.955718    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:57.045890 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:57.045856    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:46:57.093990 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:57.093880    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:58.699663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:58.699581    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:58.830725 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:58.830694    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:58.993590 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:58.993459    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:59.326321 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:59.326287    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"
Oct 30 12:46:59.693549 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:59.691938    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:59.826822 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:46:59.826789    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:00.761878 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.761845    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"]
Oct 30 12:47:00.762239 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.762227    4522 cpu_manager.go:414] "containerMap: removing container" podUID="9b7f9517-46e8-4254-bb3b-a6995326049a" containerName="payload-pre-processing"
Oct 30 12:47:00.762299 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.762240    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7f9517-46e8-4254-bb3b-a6995326049a" containerName="payload-pre-processing"
Oct 30 12:47:00.762336 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.762314    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7f9517-46e8-4254-bb3b-a6995326049a" containerName="payload-pre-processing"
Oct 30 12:47:00.762336 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.762327    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f227556-8b10-40b3-aa88-cf825443e409" containerName="istio-proxy"
Oct 30 12:47:00.783401 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.779580    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:00.802840 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.796878    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"]
Oct 30 12:47:00.851844 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.851810    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-c81f42-serving-cert\""
Oct 30 12:47:00.860152 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.857510    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:00.865528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.861019    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjv24\" (UniqueName: \"kubernetes.io/projected/186f443b-4af4-40dd-ad37-71be0c18ebf4-kube-api-access-vjv24\") pod \"maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:00.865528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.861090    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:00.865528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.861160    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186f443b-4af4-40dd-ad37-71be0c18ebf4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:00.865528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.861193    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-metrics-certs\") pod \"maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:00.957940 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.957903    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:00.962137 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.962114    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186f443b-4af4-40dd-ad37-71be0c18ebf4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:00.962232 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.962152    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-metrics-certs\") pod \"maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:00.962232 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.962223    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjv24\" (UniqueName: \"kubernetes.io/projected/186f443b-4af4-40dd-ad37-71be0c18ebf4-kube-api-access-vjv24\") pod \"maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:00.962306 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.962278    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:00.962464 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:47:00.962444    4522 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-serving-cert: secret "maas-api-e2e-worker-w3-c81f42-serving-cert" not found
Oct 30 12:47:00.962537 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:47:00.962529    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-maas-api-tls podName:186f443b-4af4-40dd-ad37-71be0c18ebf4 nodeName:}" failed. No retries permitted until 2026-10-30 12:47:01.462510172 +0000 UTC m=+665.277836644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-maas-api-tls") pod "maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544" (UID: "186f443b-4af4-40dd-ad37-71be0c18ebf4") : secret "maas-api-e2e-worker-w3-c81f42-serving-cert" not found
Oct 30 12:47:00.962941 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.962915    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186f443b-4af4-40dd-ad37-71be0c18ebf4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:00.964965 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.964941    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-metrics-certs\") pod \"maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:00.998105 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:00.998068    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjv24\" (UniqueName: \"kubernetes.io/projected/186f443b-4af4-40dd-ad37-71be0c18ebf4-kube-api-access-vjv24\") pod \"maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:01.212635 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:01.212593    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:01.487474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:01.485022    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:01.493230 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:01.493192    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:01.533041 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:01.533005    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:01.853234 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:01.853206    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"]
Oct 30 12:47:01.854295 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:47:01.854270    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186f443b_4af4_40dd_ad37_71be0c18ebf4.slice/crio-25fac7042e8940ec10d9caaa337a97887d64ca187d8d621f0a7eaceb1ec14615 WatchSource:0}: Error finding container 25fac7042e8940ec10d9caaa337a97887d64ca187d8d621f0a7eaceb1ec14615: Status 404 returned error can't find the container with id 25fac7042e8940ec10d9caaa337a97887d64ca187d8d621f0a7eaceb1ec14615
Oct 30 12:47:02.516804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:02.516711    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544" event={"ID":"186f443b-4af4-40dd-ad37-71be0c18ebf4","Type":"ContainerStarted","Data":"2df051703d6bcdb36340d0e433b2ff22545fcb9342a1a4bcfa90e53c431c5720"}
Oct 30 12:47:02.516804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:02.516782    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544" event={"ID":"186f443b-4af4-40dd-ad37-71be0c18ebf4","Type":"ContainerStarted","Data":"25fac7042e8940ec10d9caaa337a97887d64ca187d8d621f0a7eaceb1ec14615"}
Oct 30 12:47:02.517009 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:02.516817    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:02.559641 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:02.559578    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544" podStartSLOduration=2.196580338 podStartE2EDuration="2.559559367s" totalImagesPullingTime="362.979029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:02.55762964 +0000 UTC m=+666.372956129" watchObservedRunningTime="2026-10-30 12:47:02.559559367 +0000 UTC m=+666.374885857"
Oct 30 12:47:02.799159 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:02.799116    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:02.957966 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:02.957884    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:03.523287 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:03.523260    4522 generic.go:356] "Generic (PLEG): container finished" podID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerID="d61531e53dc65dbe5434cb28f9413bd502ceff2b7e1cd96f0a8e2a9166657fd6" exitCode=0
Oct 30 12:47:03.523531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:03.523340    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" event={"ID":"8e76d3f0-c1f4-4458-be8d-8d6922001217","Type":"ContainerDied","Data":"d61531e53dc65dbe5434cb28f9413bd502ceff2b7e1cd96f0a8e2a9166657fd6"}
Oct 30 12:47:03.523757 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:03.523740    4522 scope.go:122] "RemoveContainer" containerID="d61531e53dc65dbe5434cb28f9413bd502ceff2b7e1cd96f0a8e2a9166657fd6"
Oct 30 12:47:03.530675 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:03.530643    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:47:03.530781 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:03.530725    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:47:03.530781 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:03.530768    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:47:04.349516 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:04.349478    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:47:04.349913 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:04.349579    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:47:04.549122 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:04.548588    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" event={"ID":"8e76d3f0-c1f4-4458-be8d-8d6922001217","Type":"ContainerStarted","Data":"2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f"}
Oct 30 12:47:04.624104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:04.624019    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:05.346276 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:05.346238    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:47:05.346924 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:05.346887    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 12:47:05.347049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:05.346942    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 12:47:05.560422 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:05.560387    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:47:06.307921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.307877    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"]
Oct 30 12:47:06.312129 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.312105    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:06.337311 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.337278    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"]
Oct 30 12:47:06.340302 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.340278    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-7a590a-serving-cert\""
Oct 30 12:47:06.347093 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.347065    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 12:47:06.347202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.347134    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 12:47:06.405864 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.405824    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:06.417191 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.411268    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-metrics-certs\") pod \"maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:06.417191 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.411341    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051d8277-bc95-461e-92aa-ebfb4008a278-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:06.417191 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.411474    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:06.417191 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.411536    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mt4x\" (UniqueName: \"kubernetes.io/projected/051d8277-bc95-461e-92aa-ebfb4008a278-kube-api-access-4mt4x\") pod \"maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:06.511081 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.510990    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:06.512259 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.512234    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:06.512411 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.512300    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mt4x\" (UniqueName: \"kubernetes.io/projected/051d8277-bc95-461e-92aa-ebfb4008a278-kube-api-access-4mt4x\") pod \"maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:06.512411 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.512375    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-metrics-certs\") pod \"maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:06.512531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.512466    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051d8277-bc95-461e-92aa-ebfb4008a278-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:06.512768 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:47:06.512742    4522 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-serving-cert: secret "maas-api-e2e-worker-w1-7a590a-serving-cert" not found
Oct 30 12:47:06.512853 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:47:06.512833    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-maas-api-tls podName:051d8277-bc95-461e-92aa-ebfb4008a278 nodeName:}" failed. No retries permitted until 2026-10-30 12:47:07.012795116 +0000 UTC m=+670.828121601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-maas-api-tls") pod "maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7" (UID: "051d8277-bc95-461e-92aa-ebfb4008a278") : secret "maas-api-e2e-worker-w1-7a590a-serving-cert" not found
Oct 30 12:47:06.513185 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.513162    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051d8277-bc95-461e-92aa-ebfb4008a278-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:06.515603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.515578    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-metrics-certs\") pod \"maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:06.534516 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.534485    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mt4x\" (UniqueName: \"kubernetes.io/projected/051d8277-bc95-461e-92aa-ebfb4008a278-kube-api-access-4mt4x\") pod \"maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:06.627154 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:06.627074    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:07.017820 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:07.017730    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:07.020114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:07.020092    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:07.225379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:07.225259    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:07.351860 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:07.346755    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 12:47:07.351860 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:07.346821    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 12:47:07.624449 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:07.621885    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"]
Oct 30 12:47:08.347378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.346821    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 12:47:08.347378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.346888    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 12:47:08.451376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.446913    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:08.556699 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.555558    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 12:47:08.618613 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.617749    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7" event={"ID":"051d8277-bc95-461e-92aa-ebfb4008a278","Type":"ContainerStarted","Data":"acbe309fd62da524c12fb5de376e6851c5c51eb0b30df4c3ed9decd89e9bbcea"}
Oct 30 12:47:08.618613 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.617818    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7" event={"ID":"051d8277-bc95-461e-92aa-ebfb4008a278","Type":"ContainerStarted","Data":"3a7fb19e9bd53ef78fe5516becaa1fb53929b593de0fc9ceed232d2040c3a416"}
Oct 30 12:47:08.618613 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.618550    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:08.646042 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.646007    4522 generic.go:356] "Generic (PLEG): container finished" podID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerID="3f884ee20ba3ffb2b95a8d49b5d9776fe6132a9bc34156ab2377b68194825847" exitCode=0
Oct 30 12:47:08.646559 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.646279    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" event={"ID":"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc","Type":"ContainerDied","Data":"3f884ee20ba3ffb2b95a8d49b5d9776fe6132a9bc34156ab2377b68194825847"}
Oct 30 12:47:08.658014 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.657988    4522 scope.go:122] "RemoveContainer" containerID="3f884ee20ba3ffb2b95a8d49b5d9776fe6132a9bc34156ab2377b68194825847"
Oct 30 12:47:08.716143 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.697378    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:47:08.716143 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.697483    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:47:08.716143 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.697535    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:47:08.716143 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.709772    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:08.813471 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:08.809505    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7" podStartSLOduration=2.181946376 podStartE2EDuration="2.809482006s" totalImagesPullingTime="627.53563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:08.67320012 +0000 UTC m=+672.488526610" watchObservedRunningTime="2026-10-30 12:47:08.809482006 +0000 UTC m=+672.624808525"
Oct 30 12:47:09.328380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:09.328133    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:47:09.358376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:09.356625    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 12:47:09.358376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:09.356676    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 12:47:09.679380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:09.678914    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" event={"ID":"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc","Type":"ContainerStarted","Data":"434e2538b51212f936908b82775fb168e469cd90f5bcddbf6b46d590545d565f"}
Oct 30 12:47:10.024591 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:10.024560    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:47:10.024951 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:10.024928    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:47:10.025123 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:10.025100    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 12:47:10.346438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:10.337303    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:10.922824 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:10.922787    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:11.030795 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:11.030756    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:47:11.031108 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:11.031079    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 12:47:11.354631 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:11.348558    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:47:11.354631 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:11.348621    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:47:11.356741 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:11.355176    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:47:11.356741 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:11.356596    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:47:11.362691 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:11.362606    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 12:47:12.039824 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:12.039779    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:47:12.040488 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:12.040441    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 12:47:12.129788 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:12.129209    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:13.025379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:13.024792    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:47:13.025379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:13.024853    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 12:47:14.023792 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:14.023758    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:47:14.024157 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:14.023821    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 12:47:14.226090 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:14.226053    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:14.577010 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:14.576971    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:14.678619 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:14.678546    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:14.781378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:14.780679    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:15.023585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:15.023554    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:47:15.023758 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:15.023612    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 12:47:15.160632 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:15.160590    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:15.288968 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:15.288837    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:15.505259 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:15.505168    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:15.724774 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:15.724688    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:15.753911 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:15.753870    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 12:47:16.030617 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:16.023043    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:47:16.030617 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:16.023100    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 12:47:16.321339 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:16.321220    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:17.028817 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:17.028784    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:47:17.029419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:17.029399    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:47:17.030031 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:17.030016    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 12:47:17.807138 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:17.807106    4522 generic.go:356] "Generic (PLEG): container finished" podID="674b4304-a144-449c-9c7b-bd026b669f74" containerID="f3c2d524d044dbc6913cdf84b7a740c31e43c2ccab9abef1f5deefd076ef2cef" exitCode=0
Oct 30 12:47:17.809551 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:17.807181    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" event={"ID":"674b4304-a144-449c-9c7b-bd026b669f74","Type":"ContainerDied","Data":"f3c2d524d044dbc6913cdf84b7a740c31e43c2ccab9abef1f5deefd076ef2cef"}
Oct 30 12:47:17.809551 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:17.807741    4522 scope.go:122] "RemoveContainer" containerID="f3c2d524d044dbc6913cdf84b7a740c31e43c2ccab9abef1f5deefd076ef2cef"
Oct 30 12:47:17.809551 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:17.809066    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:47:17.809551 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:17.809128    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:47:17.809551 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:17.809160    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:47:18.165643 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.165549    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"]
Oct 30 12:47:18.169598 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.169576    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.185033 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.185012    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-105228\"/\"openshift-service-ca.crt\""
Oct 30 12:47:18.185484 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.185464    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-105228\"/\"kube-root-ca.crt\""
Oct 30 12:47:18.187426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.187405    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"]
Oct 30 12:47:18.196640 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.196610    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-105228\"/\"simulator-w2-105228-kserve-self-signed-certs\""
Oct 30 12:47:18.196819 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.196771    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-105228\"/\"default-dockercfg-q265w\""
Oct 30 12:47:18.214506 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.214479    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:18.231608 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.231577    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dpdp\" (UniqueName: \"kubernetes.io/projected/03285671-c2a3-4503-accb-e6e3b41a16a5-kube-api-access-9dpdp\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.231762 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.231656    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-tmp-dir\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.231762 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.231693    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03285671-c2a3-4503-accb-e6e3b41a16a5-tls-certs\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.231762 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.231741    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-model-cache\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.231898 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.231883    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-home\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.232001 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.231985    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-dshm\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.307524 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.307488    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:18.332590 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.332559    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-dshm\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.332786 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.332617    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dpdp\" (UniqueName: \"kubernetes.io/projected/03285671-c2a3-4503-accb-e6e3b41a16a5-kube-api-access-9dpdp\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.332786 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.332644    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-tmp-dir\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.332786 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.332678    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03285671-c2a3-4503-accb-e6e3b41a16a5-tls-certs\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.332786 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.332734    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-model-cache\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.333006 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.332787    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-home\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.333115 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.333094    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-model-cache\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.333172 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.333131    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-tmp-dir\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.333227 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.333186    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-home\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.334956 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.334929    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-dshm\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.335141 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.335127    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03285671-c2a3-4503-accb-e6e3b41a16a5-tls-certs\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.351236 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.351211    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dpdp\" (UniqueName: \"kubernetes.io/projected/03285671-c2a3-4503-accb-e6e3b41a16a5-kube-api-access-9dpdp\") pod \"simulator-w2-105228-kserve-66d756bcfc-7kfnf\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") " pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.498449 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.497948    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:18.813543 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.813506    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" event={"ID":"674b4304-a144-449c-9c7b-bd026b669f74","Type":"ContainerStarted","Data":"3547337f18ddfc64ee089a78195d25149a5bd7e4afe6c8cd67d89f51abbc0381"}
Oct 30 12:47:18.815028 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.815007    4522 generic.go:356] "Generic (PLEG): container finished" podID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerID="21dac7f2de25108ea47a5b9d35144a09b50a83a933a16cbaebbaf5fdf305bb2a" exitCode=0
Oct 30 12:47:18.815135 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.815074    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" event={"ID":"e726d2a8-8f05-471d-a7d1-2fa2c8df6550","Type":"ContainerDied","Data":"21dac7f2de25108ea47a5b9d35144a09b50a83a933a16cbaebbaf5fdf305bb2a"}
Oct 30 12:47:18.815498 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.815484    4522 scope.go:122] "RemoveContainer" containerID="21dac7f2de25108ea47a5b9d35144a09b50a83a933a16cbaebbaf5fdf305bb2a"
Oct 30 12:47:18.816550 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.816526    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:47:18.816606 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.816586    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:47:18.816653 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.816615    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:47:18.847301 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:18.847277    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"]
Oct 30 12:47:18.853635 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:47:18.853597    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03285671_c2a3_4503_accb_e6e3b41a16a5.slice/crio-2f522986ffe064e3bb0ab3f378ec22d3cecac422a31f235da721be8c95168545 WatchSource:0}: Error finding container 2f522986ffe064e3bb0ab3f378ec22d3cecac422a31f235da721be8c95168545: Status 404 returned error can't find the container with id 2f522986ffe064e3bb0ab3f378ec22d3cecac422a31f235da721be8c95168545
Oct 30 12:47:19.292815 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.292777    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:47:19.293164 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.293140    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 12:47:19.293338 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.293312    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 12:47:19.837076 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.837038    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"]
Oct 30 12:47:19.845715 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.845686    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"]
Oct 30 12:47:19.846042 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.846023    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:19.853711 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.853675    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:19.855850 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.855817    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" event={"ID":"e726d2a8-8f05-471d-a7d1-2fa2c8df6550","Type":"ContainerStarted","Data":"4bb9309b2a0bbbe917d52453399acf4edbd4c3981c09bddaf397c164e7bb4916"}
Oct 30 12:47:19.857780 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.857751    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" event={"ID":"03285671-c2a3-4503-accb-e6e3b41a16a5","Type":"ContainerStarted","Data":"dafd011a73ad621a1c513d23306b5aec503ee21fa493b0c27e4ed383fef1b42b"}
Oct 30 12:47:19.857891 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.857789    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" event={"ID":"03285671-c2a3-4503-accb-e6e3b41a16a5","Type":"ContainerStarted","Data":"2f522986ffe064e3bb0ab3f378ec22d3cecac422a31f235da721be8c95168545"}
Oct 30 12:47:19.888931 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.888901    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c81f42\"/\"default-dockercfg-jxlsf\""
Oct 30 12:47:19.889390 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.889372    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c81f42\"/\"simulator-w3-c81f42-kserve-self-signed-certs\""
Oct 30 12:47:19.889647 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.889632    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c81f42\"/\"openshift-service-ca.crt\""
Oct 30 12:47:19.889904 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.889881    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c81f42\"/\"kube-root-ca.crt\""
Oct 30 12:47:19.943265 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.943211    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:19.960129 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.960077    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podStartSLOduration=1.9600641319999998 podStartE2EDuration="1.960064132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:19.959438301 +0000 UTC m=+683.774764789" watchObservedRunningTime="2026-10-30 12:47:19.960064132 +0000 UTC m=+683.775390650"
Oct 30 12:47:19.999374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.998396    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tmp-dir\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:19.999374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.998455    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-dshm\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:19.999374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.998502    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tls-certs\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:19.999374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.998680    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vsnv\" (UniqueName: \"kubernetes.io/projected/2423ab7f-0469-463d-9aaf-b17895f4c4a6-kube-api-access-6vsnv\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:19.999374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.998763    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-model-cache\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:19.999374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:19.999009    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-home\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.103430 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.102940    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-home\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.103430 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.103261    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tmp-dir\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.103430 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.103319    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-dshm\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.103738 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.103442    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tls-certs\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.103738 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.103703    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vsnv\" (UniqueName: \"kubernetes.io/projected/2423ab7f-0469-463d-9aaf-b17895f4c4a6-kube-api-access-6vsnv\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.107371 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.103900    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-model-cache\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.107371 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.104619    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-model-cache\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.107371 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.104910    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-home\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.107371 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.105198    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tmp-dir\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.119376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.116183    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tls-certs\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.147925 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.147889    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vsnv\" (UniqueName: \"kubernetes.io/projected/2423ab7f-0469-463d-9aaf-b17895f4c4a6-kube-api-access-6vsnv\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.159374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.157316    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-dshm\") pod \"simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") " pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.168610 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.168571    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:20.295394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.294767    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 12:47:20.295394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.294829    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 12:47:20.503436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.502455    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:20.597493 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.597437    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"]
Oct 30 12:47:20.843550 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.843501    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:20.851024 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.850995    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:47:20.851174 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.851090    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 12:47:20.851174 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.851122    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 12:47:20.864578 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.864535    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" event={"ID":"2423ab7f-0469-463d-9aaf-b17895f4c4a6","Type":"ContainerStarted","Data":"2a389ea1e14df53b9870ac7fad2b3bea73c8076fe00bec3028236a5f8dbc6f19"}
Oct 30 12:47:20.864746 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:20.864589    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" event={"ID":"2423ab7f-0469-463d-9aaf-b17895f4c4a6","Type":"ContainerStarted","Data":"2a3a9b18681f11fa4c08e486238492ab181ed7f73e4d98604293477c18b20939"}
Oct 30 12:47:21.148789 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:47:21.148703    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:47:21.176378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:21.171464    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:21.298376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:21.297746    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 12:47:21.298376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:21.297806    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 12:47:21.421426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:21.416826    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:21.529019 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:21.528989    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:21.646378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:21.635794    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:21.850731 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:21.850694    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 12:47:21.851148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:21.850745    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 12:47:21.912377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:21.900698    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:22.297376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:22.292719    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 12:47:22.297376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:22.292777    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 12:47:22.851052 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:22.851018    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 12:47:22.851465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:22.851069    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 12:47:23.296375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:23.292711    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 12:47:23.296375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:23.292773    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 12:47:23.745568 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:23.745489    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:23.850814 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:23.850778    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 12:47:23.851088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:23.851061    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 12:47:24.347996 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:24.347968    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:47:24.366142 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:24.366111    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:47:24.436075 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:24.436023    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podStartSLOduration=5.43601033 podStartE2EDuration="5.43601033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:20.927152081 +0000 UTC m=+684.742478595" watchObservedRunningTime="2026-10-30 12:47:24.43601033 +0000 UTC m=+688.251336818"
Oct 30 12:47:24.850574 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:24.850535    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 12:47:24.850760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:24.850598    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 12:47:24.918768 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:24.918739    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 12:47:25.403205 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:25.403172    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:25.828374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:25.828318    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:47:25.854498 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:25.854473    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:47:25.854817 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:25.854792    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:47:25.855526 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:25.855510    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 12:47:27.263137 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:27.263101    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:28.498901 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:28.498857    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:28.511480 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:28.511450    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:28.948674 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:28.948646    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:47:29.101913 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:29.101875    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:30.171239 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:30.171210    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:30.183519 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:30.183494    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:30.303294 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:30.303264    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:30.919196 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:30.919164    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:30.958133 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:30.958105    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:47:31.987795 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:31.987752    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:32.091147 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.091114    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:32.208565 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.208526    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:32.839810 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.839774    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:32.890057 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.890017    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"]
Oct 30 12:47:32.896819 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.896792    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:32.912211 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.912186    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-53dcb1\"/\"kube-root-ca.crt\""
Oct 30 12:47:32.912693 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.912668    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"]
Oct 30 12:47:32.913637 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.913610    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b974f0-872d-457a-b442-71e652a012cc-tls-certs\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:32.913819 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.913791    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-home\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:32.913933 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.913837    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-model-cache\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:32.914012 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.913994    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-dshm\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:32.914071 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.914053    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvb89\" (UniqueName: \"kubernetes.io/projected/76b974f0-872d-457a-b442-71e652a012cc-kube-api-access-pvb89\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:32.914160 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.914135    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-tmp-dir\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:32.919609 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.919587    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-53dcb1\"/\"openshift-service-ca.crt\""
Oct 30 12:47:32.919727 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.919629    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-53dcb1\"/\"default-dockercfg-v2ppp\""
Oct 30 12:47:32.919727 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.919663    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-53dcb1\"/\"simulator-w4-53dcb1-kserve-self-signed-certs\""
Oct 30 12:47:32.940669 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:32.940641    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:33.015821 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.015790    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-dshm\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:33.016807 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.016777    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvb89\" (UniqueName: \"kubernetes.io/projected/76b974f0-872d-457a-b442-71e652a012cc-kube-api-access-pvb89\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:33.016935 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.016818    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-tmp-dir\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:33.016935 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.016928    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b974f0-872d-457a-b442-71e652a012cc-tls-certs\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:33.017050 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.016966    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-home\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:33.017050 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.016995    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-model-cache\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:33.017263 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.017237    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-tmp-dir\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:33.017337 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.017287    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-model-cache\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:33.017507 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.017466    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-home\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:33.019913 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.019888    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b974f0-872d-457a-b442-71e652a012cc-tls-certs\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:33.021764 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.021741    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-dshm\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:33.040548 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.040526    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvb89\" (UniqueName: \"kubernetes.io/projected/76b974f0-872d-457a-b442-71e652a012cc-kube-api-access-pvb89\") pod \"simulator-w4-53dcb1-kserve-67f786dcf-fsr2n\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") " pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:33.213753 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.213661    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:33.515593 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:47:33.515560    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76b974f0_872d_457a_b442_71e652a012cc.slice/crio-5f7dd620cc66812ef5d112c14fe62ffaffc62015a490bc8be51c9811642dfd1c WatchSource:0}: Error finding container 5f7dd620cc66812ef5d112c14fe62ffaffc62015a490bc8be51c9811642dfd1c: Status 404 returned error can't find the container with id 5f7dd620cc66812ef5d112c14fe62ffaffc62015a490bc8be51c9811642dfd1c
Oct 30 12:47:33.515845 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.515826    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"]
Oct 30 12:47:33.964365 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.964319    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" event={"ID":"76b974f0-872d-457a-b442-71e652a012cc","Type":"ContainerStarted","Data":"5f081f824c8cac562cfe1f79d493132dca3d84b12983a511986e28b363f8a861"}
Oct 30 12:47:33.964557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:33.964385    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" event={"ID":"76b974f0-872d-457a-b442-71e652a012cc","Type":"ContainerStarted","Data":"5f7dd620cc66812ef5d112c14fe62ffaffc62015a490bc8be51c9811642dfd1c"}
Oct 30 12:47:34.007922 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:34.007875    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podStartSLOduration=2.007861962 podStartE2EDuration="2.007861962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:34.005529786 +0000 UTC m=+697.820856278" watchObservedRunningTime="2026-10-30 12:47:34.007861962 +0000 UTC m=+697.823188451"
Oct 30 12:47:34.048487 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:34.048462    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:47:34.213920 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:34.213888    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:34.936193 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:34.936157    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:36.613220 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:36.613163    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:38.320766 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:38.320731    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:40.010476 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:40.010438    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:41.937786 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:41.937739    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:42.728805 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:42.728767    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:43.214780 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:43.214746    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:43.227793 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:43.227770    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:43.287333 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:43.287304    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:47:43.832596 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:43.832564    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:44.060310 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:44.060281    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:47:45.611204 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:45.611172    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:47.151259 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:47.151205    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:47.571731 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:47.571696    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:49.255863 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:49.255827    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:50.188807 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.188768    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"]
Oct 30 12:47:50.192535 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.192515    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.213254 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.213233    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"]
Oct 30 12:47:50.222651 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.222626    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c81f42\"/\"premium-w3-c81f42-kserve-self-signed-certs\""
Oct 30 12:47:50.269434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.269413    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:50.341251 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.341221    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-tmp-dir\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.341377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.341275    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-dshm\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.341446 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.341344    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchfr\" (UniqueName: \"kubernetes.io/projected/f59f4327-f9c2-4445-97b6-63a99043d625-kube-api-access-lchfr\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.341496 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.341443    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-model-cache\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.341553 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.341535    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-home\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.341657 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.341637    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f59f4327-f9c2-4445-97b6-63a99043d625-tls-certs\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.443612 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.443532    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f59f4327-f9c2-4445-97b6-63a99043d625-tls-certs\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.443855 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.443836    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-tmp-dir\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.443984 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.443964    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-dshm\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.444073 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.443997    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchfr\" (UniqueName: \"kubernetes.io/projected/f59f4327-f9c2-4445-97b6-63a99043d625-kube-api-access-lchfr\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.444073 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.444063    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-model-cache\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.444173 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.444128    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-home\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.444948 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.444891    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-home\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.444948 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.444921    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-tmp-dir\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.445118 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.444951    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-model-cache\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.446727 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.446701    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-dshm\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.447715 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.447692    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f59f4327-f9c2-4445-97b6-63a99043d625-tls-certs\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.463561 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.463539    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchfr\" (UniqueName: \"kubernetes.io/projected/f59f4327-f9c2-4445-97b6-63a99043d625-kube-api-access-lchfr\") pod \"premium-w3-c81f42-kserve-58f455bcdf-vvxvt\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") " pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.506081 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.506058    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:50.806580 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:50.806554    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"]
Oct 30 12:47:50.806879 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:47:50.806857    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf59f4327_f9c2_4445_97b6_63a99043d625.slice/crio-b56ae8666115a827844578dc6550dc021a8eed8c9e5b8fbd841e0fd8ea29b3bc WatchSource:0}: Error finding container b56ae8666115a827844578dc6550dc021a8eed8c9e5b8fbd841e0fd8ea29b3bc: Status 404 returned error can't find the container with id b56ae8666115a827844578dc6550dc021a8eed8c9e5b8fbd841e0fd8ea29b3bc
Oct 30 12:47:51.023892 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:51.023859    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:51.066503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:51.066429    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:47:51.114727 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:51.114688    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" event={"ID":"f59f4327-f9c2-4445-97b6-63a99043d625","Type":"ContainerStarted","Data":"5c1c9d2eaed8f08aebd0f250cdee900ce150c7ce91ea102e4bdfd84f85663a2a"}
Oct 30 12:47:51.114933 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:51.114740    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" event={"ID":"f59f4327-f9c2-4445-97b6-63a99043d625","Type":"ContainerStarted","Data":"b56ae8666115a827844578dc6550dc021a8eed8c9e5b8fbd841e0fd8ea29b3bc"}
Oct 30 12:47:51.143455 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:47:51.143421    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:47:51.161549 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:51.161497    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podStartSLOduration=1.161481771 podStartE2EDuration="1.161481771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:51.159803556 +0000 UTC m=+714.975130045" watchObservedRunningTime="2026-10-30 12:47:51.161481771 +0000 UTC m=+714.976808308"
Oct 30 12:47:51.506805 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:51.506736    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:47:52.985306 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:52.985099    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:54.808125 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:54.808093    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:56.359122 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:56.359087    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:47:56.632135 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:56.632058    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:58.553789 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:58.553754    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:58.756425 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:47:58.754878    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:00.322567 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:00.322533    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:00.507197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:00.507149    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:48:00.519604 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:00.519577    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:48:01.203275 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:01.203245    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:48:01.737434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:01.737401    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:48:02.126949 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:02.126917    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:04.033280 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:04.033240    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:05.765489 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:05.765453    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:06.047128 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.047098    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"]
Oct 30 12:48:06.052246 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.052218    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.069934 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.069910    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"]
Oct 30 12:48:06.079548 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.079522    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-105228\"/\"premium-w2-105228-kserve-self-signed-certs\""
Oct 30 12:48:06.103736 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.103707    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:06.165820 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.165792    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-home\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.165934 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.165839    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27xr\" (UniqueName: \"kubernetes.io/projected/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-kube-api-access-x27xr\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.165934 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.165862    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-model-cache\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.166038 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.166019    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tmp-dir\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.166087 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.166051    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-dshm\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.166154 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.166133    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tls-certs\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.270395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.268988    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tmp-dir\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.270395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.269043    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-dshm\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.270395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.269094    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tls-certs\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.270395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.269199    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-home\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.270395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.269253    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27xr\" (UniqueName: \"kubernetes.io/projected/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-kube-api-access-x27xr\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.270395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.269288    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-model-cache\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.270395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.269800    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-model-cache\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.270395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.270051    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tmp-dir\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.271257 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.271233    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-home\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.283455 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.283146    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tls-certs\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.298697 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.298621    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-dshm\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.307647 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.307607    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27xr\" (UniqueName: \"kubernetes.io/projected/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-kube-api-access-x27xr\") pod \"premium-w2-105228-kserve-7bf6cb7f6d-bwvqw\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") " pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.367067 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.367035    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:06.677650 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:06.677620    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"]
Oct 30 12:48:06.677963 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:48:06.677942    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3fda1a0_5841_4eaa_9621_1fe8b5165b12.slice/crio-417e93e9f078c3047bc1ec3e538c2f4f98aa66e9fa29e1bd5351d3d2a2cac394 WatchSource:0}: Error finding container 417e93e9f078c3047bc1ec3e538c2f4f98aa66e9fa29e1bd5351d3d2a2cac394: Status 404 returned error can't find the container with id 417e93e9f078c3047bc1ec3e538c2f4f98aa66e9fa29e1bd5351d3d2a2cac394
Oct 30 12:48:07.296178 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:07.296129    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" event={"ID":"f3fda1a0-5841-4eaa-9621-1fe8b5165b12","Type":"ContainerStarted","Data":"c491c293b178a495a60bf678dff8a1270ad5f7017b9cee444c4f815b817ea4c4"}
Oct 30 12:48:07.296178 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:07.296176    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" event={"ID":"f3fda1a0-5841-4eaa-9621-1fe8b5165b12","Type":"ContainerStarted","Data":"417e93e9f078c3047bc1ec3e538c2f4f98aa66e9fa29e1bd5351d3d2a2cac394"}
Oct 30 12:48:07.367659 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:07.367628    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:07.421305 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:07.421272    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:09.355207 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:09.355157    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:11.216931 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:11.216895    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:13.037015 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.036979    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:13.652259 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.652208    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podStartSLOduration=7.652194558 podStartE2EDuration="7.652194558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:07.33646596 +0000 UTC m=+731.151792450" watchObservedRunningTime="2026-10-30 12:48:13.652194558 +0000 UTC m=+737.467521097"
Oct 30 12:48:13.652643 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.652625    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"]
Oct 30 12:48:13.656250 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.656234    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.676626 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.676584    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"]
Oct 30 12:48:13.685773 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.685750    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-53dcb1\"/\"premium-w4-53dcb1-kserve-self-signed-certs\""
Oct 30 12:48:13.692391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.692368    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:13.763215 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.763191    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-model-cache\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.763312 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.763218    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-home\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.763312 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.763248    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f886013f-7c33-4731-84e8-4dcff7f18fd3-tls-certs\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.763468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.763447    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-tmp-dir\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.763512 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.763481    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26fpf\" (UniqueName: \"kubernetes.io/projected/f886013f-7c33-4731-84e8-4dcff7f18fd3-kube-api-access-26fpf\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.763512 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.763503    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-dshm\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.868374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.864331    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f886013f-7c33-4731-84e8-4dcff7f18fd3-tls-certs\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.868374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.864483    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-tmp-dir\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.868374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.864516    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fpf\" (UniqueName: \"kubernetes.io/projected/f886013f-7c33-4731-84e8-4dcff7f18fd3-kube-api-access-26fpf\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.868374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.864546    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-dshm\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.868374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.864631    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-model-cache\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.868374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.864660    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-home\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.868374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.865151    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-home\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.868374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.866968    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-tmp-dir\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.868374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.867931    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-model-cache\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.876769 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.876738    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f886013f-7c33-4731-84e8-4dcff7f18fd3-tls-certs\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.885409 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.883021    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-dshm\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.904417 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.901864    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fpf\" (UniqueName: \"kubernetes.io/projected/f886013f-7c33-4731-84e8-4dcff7f18fd3-kube-api-access-26fpf\") pod \"premium-w4-53dcb1-kserve-654f454598-bngll\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") " pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:13.971909 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:13.971873    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:14.209810 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:14.209743    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:48:14.273160 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:48:14.273133    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf886013f_7c33_4731_84e8_4dcff7f18fd3.slice/crio-59296e9efe7cca1289cf05d31e01d33a2337df582fa72065f5b1761db86b2c62 WatchSource:0}: Error finding container 59296e9efe7cca1289cf05d31e01d33a2337df582fa72065f5b1761db86b2c62: Status 404 returned error can't find the container with id 59296e9efe7cca1289cf05d31e01d33a2337df582fa72065f5b1761db86b2c62
Oct 30 12:48:14.273301 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:14.273222    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"]
Oct 30 12:48:14.333919 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:14.333892    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" event={"ID":"f886013f-7c33-4731-84e8-4dcff7f18fd3","Type":"ContainerStarted","Data":"2be2447f716cab56852b14f7e3e5e3882c73fdf42ffc53caa75b3fda08dfa752"}
Oct 30 12:48:14.333997 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:14.333931    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" event={"ID":"f886013f-7c33-4731-84e8-4dcff7f18fd3","Type":"ContainerStarted","Data":"59296e9efe7cca1289cf05d31e01d33a2337df582fa72065f5b1761db86b2c62"}
Oct 30 12:48:14.374655 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:14.374614    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podStartSLOduration=1.374601172 podStartE2EDuration="1.374601172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:14.373611573 +0000 UTC m=+738.188938062" watchObservedRunningTime="2026-10-30 12:48:14.374601172 +0000 UTC m=+738.189927660"
Oct 30 12:48:14.819533 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:14.819492    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:14.972600 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:14.972566    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:16.368719 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:16.368157    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:16.404382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:16.395019    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:16.775386 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:16.775270    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:17.367797 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:17.367762    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:48:18.608932 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:18.608897    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:20.421314 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:20.421283    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:20.771486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:20.771455    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:21.060197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:21.060111    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:48:21.133639 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:48:21.133607    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:48:22.252627 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.252593    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:22.660398 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.660339    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"]
Oct 30 12:48:22.680377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.666336    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.684798 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.684761    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"]
Oct 30 12:48:22.693068 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.693037    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7a590a-openshift-default-dockercfg-4zgs6\""
Oct 30 12:48:22.729945 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.729911    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a4b170-e0c7-4b26-8817-aa969962c106-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.730173 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.730150    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.730334 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.730316    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.730657 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.730632    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.730879 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.730849    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.731063 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.731044    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.731243 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.731225    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.731393 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.731376    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckhwx\" (UniqueName: \"kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-kube-api-access-ckhwx\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.731665 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.731641    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.832827 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.832791    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.833020 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.832840    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.833020 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.832891    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.833020 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.832916    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckhwx\" (UniqueName: \"kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-kube-api-access-ckhwx\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.833020 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.832936    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.833020 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.832968    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a4b170-e0c7-4b26-8817-aa969962c106-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.833020 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.832988    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.833388 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.833042    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.833388 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.833083    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.833388 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.833344    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.833514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.833397    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.833514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.833448    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.833684 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.833660    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.833821 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.833802    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a4b170-e0c7-4b26-8817-aa969962c106-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.835305 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.835283    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.835514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.835496    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.854886 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.854854    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:22.854998 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:22.854860    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckhwx\" (UniqueName: \"kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-kube-api-access-ckhwx\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:23.035708 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:23.035671    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:23.380082 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:23.380039    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"]
Oct 30 12:48:23.432624 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:23.429175    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:23.432624 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:23.429269    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:23.432624 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:23.429311    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:23.438856 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:23.438736    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2" event={"ID":"c3a4b170-e0c7-4b26-8817-aa969962c106","Type":"ContainerStarted","Data":"f88b2138b7fce40d0550e4d45fa026a2860342e2193ccac4ac37a70ebe4de408"}
Oct 30 12:48:23.983378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:23.980621    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:24.046285 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:24.045980    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:24.365159 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:24.365126    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:24.445403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:24.445366    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2" event={"ID":"c3a4b170-e0c7-4b26-8817-aa969962c106","Type":"ContainerStarted","Data":"4ad43cfedf483cf7ce66dfe8e9a830907693bc2000a3bfc501a747188a1928f2"}
Oct 30 12:48:24.456807 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:24.456781    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:48:24.496527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:24.496465    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2" podStartSLOduration=2.496445073 podStartE2EDuration="2.496445073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:24.492887857 +0000 UTC m=+748.308214348" watchObservedRunningTime="2026-10-30 12:48:24.496445073 +0000 UTC m=+748.311771582"
Oct 30 12:48:25.036472 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:25.036433    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:25.038053 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:25.038023    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:48:25.038181 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:25.038075    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2" podUID="c3a4b170-e0c7-4b26-8817-aa969962c106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 12:48:26.036362 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:26.036315    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:48:26.036736 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:26.036392    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2" podUID="c3a4b170-e0c7-4b26-8817-aa969962c106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 12:48:26.361182 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:26.361103    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:27.036390 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:27.036330    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:48:27.036848 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:27.036448    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2" podUID="c3a4b170-e0c7-4b26-8817-aa969962c106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 12:48:28.036220 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:28.036186    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:48:28.036394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:28.036245    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2" podUID="c3a4b170-e0c7-4b26-8817-aa969962c106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 12:48:28.358662 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:28.358562    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:28.914500 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:28.914464    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:48:29.036604 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:29.036572    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:48:29.036771 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:29.036622    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2" podUID="c3a4b170-e0c7-4b26-8817-aa969962c106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 12:48:30.086284 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:30.086252    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:30.086855 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:30.086833    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:30.088144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:30.088121    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 12:48:30.966330 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:30.966288    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:33.687334 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:33.687298    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:35.631202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:35.631137    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:36.144562 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:36.144523    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:37.321883 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.321852    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"]
Oct 30 12:48:37.326863 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.326839    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.356772 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.356736    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"]
Oct 30 12:48:37.380916 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.380883    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.381049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.380927    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.381049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.380982    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.381177 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.381082    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.381237 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.381174    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.381237 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.381201    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st846\" (UniqueName: \"kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-kube-api-access-st846\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.381342 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.381310    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.381447 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.381376    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.381447 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.381417    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.469960 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.469922    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:37.482855 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.482834    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.482962 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.482863    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.482962 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.482895    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.482962 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.482951    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.483103 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.482975    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.483103 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.482991    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.483103 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.483011    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.483421 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.483394    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.483534 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.483472    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.483534 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.483499    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.483632 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.483534    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.483632 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.483537    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.483632 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.483596    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st846\" (UniqueName: \"kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-kube-api-access-st846\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.484438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.484414    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.485237 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.485208    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.485614 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.485597    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.501121 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.501099    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.501526 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.501509    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st846\" (UniqueName: \"kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-kube-api-access-st846\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.640230 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.640148    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:37.955832 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.955805    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"]
Oct 30 12:48:37.956956 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:48:37.956928    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8a14d3_f3ed_43ec_aed2_8017b4f195f6.slice/crio-0fe59e0cc347c43b768d248777a98b344ca6555aba3259b5dda70c92004473ff WatchSource:0}: Error finding container 0fe59e0cc347c43b768d248777a98b344ca6555aba3259b5dda70c92004473ff: Status 404 returned error can't find the container with id 0fe59e0cc347c43b768d248777a98b344ca6555aba3259b5dda70c92004473ff
Oct 30 12:48:37.959409 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.959373    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:37.959516 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.959453    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:37.959516 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:37.959497    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:38.565756 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:38.565667    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" event={"ID":"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6","Type":"ContainerStarted","Data":"40f0e7b9f5e84eec9cc6d8e2030b9aecdb7f855c67aac71396e707f42c7d091b"}
Oct 30 12:48:38.566167 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:38.565758    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" event={"ID":"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6","Type":"ContainerStarted","Data":"0fe59e0cc347c43b768d248777a98b344ca6555aba3259b5dda70c92004473ff"}
Oct 30 12:48:38.630289 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:38.630244    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podStartSLOduration=1.63023066 podStartE2EDuration="1.63023066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:38.617099486 +0000 UTC m=+762.432425975" watchObservedRunningTime="2026-10-30 12:48:38.63023066 +0000 UTC m=+762.445557149"
Oct 30 12:48:38.640802 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:38.640780    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:39.259841 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:39.259789    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:39.642943 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:39.642911    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 12:48:39.643336 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:39.642972    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 12:48:40.454062 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:40.454022    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:48:40.641151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:40.641119    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 12:48:40.641311 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:40.641177    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 12:48:41.261278 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.261240    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:41.614395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.614102    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"]
Oct 30 12:48:41.618545 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.618122    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.638218 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.638196    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.638368 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.638284    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.638368 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.638340    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45b4a4c-94e4-4897-8523-0e446e994bea-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.638480 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.638418    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.638562 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.638532    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.638619 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.638582    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.638701 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.638680    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdv8q\" (UniqueName: \"kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-kube-api-access-rdv8q\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.638777 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.638757    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.638842 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.638824    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.640735 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.640702    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 12:48:41.640856 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.640749    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 12:48:41.701526 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.701491    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"]
Oct 30 12:48:41.755402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.754012    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.755402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.754062    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45b4a4c-94e4-4897-8523-0e446e994bea-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.755402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.754107    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.755402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.754164    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.755402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.754191    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.755402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.754266    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdv8q\" (UniqueName: \"kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-kube-api-access-rdv8q\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.755402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.754318    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.755402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.754379    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.755402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.754419    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.755402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.754894    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.755402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.755170    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.756220 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.755838    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45b4a4c-94e4-4897-8523-0e446e994bea-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.761247 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.761211    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.762295 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.761755    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.839426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.800028    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.839426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.809087    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.861394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.855792    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdv8q\" (UniqueName: \"kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-kube-api-access-rdv8q\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.866768 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.866667    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:41.946718 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:41.943582    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:42.652415 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:42.646833    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 12:48:42.652415 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:42.646913    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 12:48:42.875493 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:42.875453    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:42.876123 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:42.876009    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:42.876292 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:42.876273    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:42.914969 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:42.914894    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"]
Oct 30 12:48:43.641299 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:43.641265    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 12:48:43.641478 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:43.641318    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 12:48:43.681377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:43.680292    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" event={"ID":"b45b4a4c-94e4-4897-8523-0e446e994bea","Type":"ContainerStarted","Data":"72626aae9e9e3a957ae9125660854ec173922e4c29576a7cf007e08f2f7f87ff"}
Oct 30 12:48:43.681377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:43.680422    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" event={"ID":"b45b4a4c-94e4-4897-8523-0e446e994bea","Type":"ContainerStarted","Data":"fbbfb00b2d89d99029f20d1bf673c3ee1a824e4089b1d83759b118587b67519e"}
Oct 30 12:48:43.694377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:43.687556    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:43.765117 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:43.764993    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" podStartSLOduration=2.764973153 podStartE2EDuration="2.764973153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:43.748239358 +0000 UTC m=+767.563565851" watchObservedRunningTime="2026-10-30 12:48:43.764973153 +0000 UTC m=+767.580299643"
Oct 30 12:48:43.946551 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:43.946472    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:44.641167 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:44.641134    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 12:48:44.641374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:44.641191    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 12:48:44.948374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:44.948259    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:48:44.948374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:44.948325    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 12:48:45.560569 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:45.560530    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:45.946789 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:45.946704    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:48:45.946789 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:45.946769    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 12:48:46.165679 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.165644    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"]
Oct 30 12:48:46.186375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.185599    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.196400 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.196375    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"]
Oct 30 12:48:46.224331 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.224276    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n4fg\" (UniqueName: \"kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-kube-api-access-7n4fg\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.224508 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.224332    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.224585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.224517    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.224649 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.224600    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.224710 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.224643    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.224710 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.224668    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.224710 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.224702    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.224876 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.224793    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.224876 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.224828    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.296510 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.295477    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:46.296869 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.296844    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:46.298721 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.298690    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 12:48:46.326334 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.326309    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.326460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.326344    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.326460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.326386    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.326460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.326408    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.326626 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.326565    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.326685 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.326631    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.326774 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.326737    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.326774 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.326751    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n4fg\" (UniqueName: \"kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-kube-api-access-7n4fg\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.326884 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.326779    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.326884 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.326842    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.326884 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.326852    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.327050 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.326915    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.327395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.327152    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.327395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.327277    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.328930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.328902    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.329466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.329449    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.346583 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.346559    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.346797 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.346774    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n4fg\" (UniqueName: \"kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-kube-api-access-7n4fg\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.530101 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.530078    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:46.865753 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:48:46.865725    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b0681f_22b7_40f0_9d42_d4efadbb44ee.slice/crio-52410049f90c60cf881758e476d01be99a41e18c994db21d7c6eed1719dd69dd WatchSource:0}: Error finding container 52410049f90c60cf881758e476d01be99a41e18c994db21d7c6eed1719dd69dd: Status 404 returned error can't find the container with id 52410049f90c60cf881758e476d01be99a41e18c994db21d7c6eed1719dd69dd
Oct 30 12:48:46.866151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.866129    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"]
Oct 30 12:48:46.892375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.889101    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:46.892375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.889190    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:46.892375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.889238    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:46.947392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.947366    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:48:46.947511 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:46.947417    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 12:48:47.275373 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:47.272821    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:47.375779 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:47.375741    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:47.730046 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:47.729951    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" event={"ID":"88b0681f-22b7-40f0-9d42-d4efadbb44ee","Type":"ContainerStarted","Data":"131154c300047adf1fe99173b9ace08037b0219a09f19cf581ee9b5b21b139d3"}
Oct 30 12:48:47.730046 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:47.730011    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" event={"ID":"88b0681f-22b7-40f0-9d42-d4efadbb44ee","Type":"ContainerStarted","Data":"52410049f90c60cf881758e476d01be99a41e18c994db21d7c6eed1719dd69dd"}
Oct 30 12:48:47.804385 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:47.803294    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podStartSLOduration=1.803274736 podStartE2EDuration="1.803274736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:47.78692502 +0000 UTC m=+771.602251502" watchObservedRunningTime="2026-10-30 12:48:47.803274736 +0000 UTC m=+771.618601225"
Oct 30 12:48:47.947440 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:47.947403    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:48:47.947626 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:47.947464    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 12:48:48.530341 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:48.530313    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:48.531630 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:48.531608    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 12:48:48.531751 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:48.531650    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 12:48:48.947309 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:48.947218    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:48:48.947309 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:48.947281    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 12:48:49.122188 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:49.122154    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:49.530731 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:49.530696    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 12:48:49.531150 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:49.530748    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 12:48:49.946813 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:49.946721    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:48:49.946813 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:49.946778    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 12:48:50.531250 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:50.531214    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 12:48:50.531647 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:50.531269    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 12:48:50.899766 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:50.899659    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:50.950986 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:50.950932    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:50.951346 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:50.951323    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:50.952068 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:50.952047    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 12:48:51.137902 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:48:51.137868    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:48:51.530516 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:51.530488    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 12:48:51.530679 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:51.530535    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 12:48:52.395401 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.395367    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"]
Oct 30 12:48:52.400509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.400479    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.416597 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.416340    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"]
Oct 30 12:48:52.523127 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.523098    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.523293 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.523148    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b9e4de-979a-4e21-9714-c6a99f26d861-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.523293 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.523227    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.523380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.523288    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.523380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.523316    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9kds\" (UniqueName: \"kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-kube-api-access-k9kds\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.523450 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.523413    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.523450 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.523434    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.523515 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.523454    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.523610 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.523593    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.531015 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.530998    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 12:48:52.531095 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.531043    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 12:48:52.624799 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.624770    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.624953 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.624816    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b9e4de-979a-4e21-9714-c6a99f26d861-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.624953 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.624850    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.624953 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.624869    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.624953 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.624885    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9kds\" (UniqueName: \"kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-kube-api-access-k9kds\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.624953 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.624948    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.625238 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.624972    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.625238 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.625001    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.625238 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.625108    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.625439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.625339    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.625439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.625379    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.625537 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.625445    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.625537 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.625471    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.625613 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.625599    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b9e4de-979a-4e21-9714-c6a99f26d861-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.627118 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.627097    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.627615 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.627588    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.644206 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.644184    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.644406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.644389    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9kds\" (UniqueName: \"kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-kube-api-access-k9kds\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:52.690406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.689971    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:52.735383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:52.731519    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:53.085276 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:48:53.085243    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b9e4de_979a_4e21_9714_c6a99f26d861.slice/crio-bdbcebfe761ec7ac9ef0994dda11e89009aa9bb79d0447f045c6afa28b6b1e83 WatchSource:0}: Error finding container bdbcebfe761ec7ac9ef0994dda11e89009aa9bb79d0447f045c6afa28b6b1e83: Status 404 returned error can't find the container with id bdbcebfe761ec7ac9ef0994dda11e89009aa9bb79d0447f045c6afa28b6b1e83
Oct 30 12:48:53.085276 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:53.085264    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"]
Oct 30 12:48:53.087738 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:53.087702    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:53.087830 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:53.087764    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:53.087830 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:53.087795    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:53.775643 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:53.775597    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" event={"ID":"14b9e4de-979a-4e21-9714-c6a99f26d861","Type":"ContainerStarted","Data":"bd55715591d3d7111a83f1013c14e871c6990a14bc22f9e788a5bef9869b2c24"}
Oct 30 12:48:53.776054 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:53.775740    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" event={"ID":"14b9e4de-979a-4e21-9714-c6a99f26d861","Type":"ContainerStarted","Data":"bdbcebfe761ec7ac9ef0994dda11e89009aa9bb79d0447f045c6afa28b6b1e83"}
Oct 30 12:48:53.785420 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:53.785392    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:53.785768 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:53.785745    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:53.786669 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:53.786649    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 12:48:53.843322 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:53.843244    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podStartSLOduration=1.843223617 podStartE2EDuration="1.843223617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:53.828549919 +0000 UTC m=+777.643876422" watchObservedRunningTime="2026-10-30 12:48:53.843223617 +0000 UTC m=+777.658550123"
Oct 30 12:48:54.541480 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:54.541440    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:54.732874 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:54.732838    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:48:54.743068 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:54.739635    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 12:48:54.743068 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:54.739695    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 12:48:55.732268 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:55.732205    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 12:48:55.732268 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:55.732264    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 12:48:55.939116 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:55.939083    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"]
Oct 30 12:48:55.943102 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:55.943079    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:55.997484 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:55.997408    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"]
Oct 30 12:48:56.099275 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.099243    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.099466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.099280    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.099466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.099302    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.099466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.099450    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.099583 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.099489    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.099583 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.099517    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.099644 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.099584    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208f0cbd-3ed2-4e90-9929-277352e907d3-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.099675 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.099641    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qzs7\" (UniqueName: \"kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-kube-api-access-2qzs7\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.099733 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.099718    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.200819 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.200787    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.200977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.200880    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.200977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.200911    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.200977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.200939    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.201140 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.200991    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.201140 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.201038    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.201140 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.201078    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.201140 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.201120    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208f0cbd-3ed2-4e90-9929-277352e907d3-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.201341 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.201158    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qzs7\" (UniqueName: \"kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-kube-api-access-2qzs7\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.201417 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.201332    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.201417 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.201389    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.201618 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.201592    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.201736 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.201719    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.202375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.202332    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208f0cbd-3ed2-4e90-9929-277352e907d3-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.203235 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.203213    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.203731 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.203712    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.220055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.220032    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.220125 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.220066    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qzs7\" (UniqueName: \"kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-kube-api-access-2qzs7\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.257935 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.257886    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:56.362090 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.362024    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:56.565289 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.565261    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"]
Oct 30 12:48:56.582613 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.582585    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.599939 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.599908    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"]
Oct 30 12:48:56.619728 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.619697    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"]
Oct 30 12:48:56.621436 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:48:56.621412    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208f0cbd_3ed2_4e90_9929_277352e907d3.slice/crio-d59874f36e001efcde35fe39ebeb56c48e7b830f5daf5b826440a50ce6c521d2 WatchSource:0}: Error finding container d59874f36e001efcde35fe39ebeb56c48e7b830f5daf5b826440a50ce6c521d2: Status 404 returned error can't find the container with id d59874f36e001efcde35fe39ebeb56c48e7b830f5daf5b826440a50ce6c521d2
Oct 30 12:48:56.626459 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.626421    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:56.626531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.626504    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:56.626568 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.626548    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:56.708186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.708155    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.708294 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.708199    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.708294 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.708223    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.708387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.708299    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.708461 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.708440    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.708500 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.708486    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.708533 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.708519    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcwmt\" (UniqueName: \"kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-kube-api-access-zcwmt\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.708575 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.708561    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.708628 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.708617    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.732741 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.732716    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 12:48:56.733045 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.732779    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 12:48:56.809956 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.809910    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.810151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.809967    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.810151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.810022    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwmt\" (UniqueName: \"kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-kube-api-access-zcwmt\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.810151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.810058    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.810151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.810143    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.810376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.810204    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.810376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.810247    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.810376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.810272    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.810376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.810306    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.810643 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.810620    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.811033 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.810808    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.811147 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.811113    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.811484 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.811459    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.811698 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.811680    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.812805 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.812779    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" event={"ID":"208f0cbd-3ed2-4e90-9929-277352e907d3","Type":"ContainerStarted","Data":"df8a2b662fd63696c6b94c72696f0ef0257670be97ae060b3b6490bf891c3d38"}
Oct 30 12:48:56.812898 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.812828    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" event={"ID":"208f0cbd-3ed2-4e90-9929-277352e907d3","Type":"ContainerStarted","Data":"d59874f36e001efcde35fe39ebeb56c48e7b830f5daf5b826440a50ce6c521d2"}
Oct 30 12:48:56.813488 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.813469    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.813940 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.813921    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.830404 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.830153    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.830614 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.830596    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcwmt\" (UniqueName: \"kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-kube-api-access-zcwmt\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:56.859761 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.859707    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" podStartSLOduration=1.859689755 podStartE2EDuration="1.859689755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:56.859216004 +0000 UTC m=+780.674542493" watchObservedRunningTime="2026-10-30 12:48:56.859689755 +0000 UTC m=+780.675016246"
Oct 30 12:48:56.921532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:56.921504    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:57.258410 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:57.258283    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:48:57.259652 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:57.259626    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 12:48:57.259724 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:57.259689    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 12:48:57.285508 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:57.285462    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"]
Oct 30 12:48:57.323309 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:48:57.323086    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6f2bb2_7f4c_4672_a201_9b822bea3687.slice/crio-17a02fe97965584ab5e181e14177604d3e3e0e0dc6975e2092a8691a1f23f3c8 WatchSource:0}: Error finding container 17a02fe97965584ab5e181e14177604d3e3e0e0dc6975e2092a8691a1f23f3c8: Status 404 returned error can't find the container with id 17a02fe97965584ab5e181e14177604d3e3e0e0dc6975e2092a8691a1f23f3c8
Oct 30 12:48:57.325462 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:57.325413    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:57.325560 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:57.325511    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:57.325606 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:57.325554    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:48:57.732804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:57.732762    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 12:48:57.733201 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:57.732815    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 12:48:57.831193 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:57.831162    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" event={"ID":"4e6f2bb2-7f4c-4672-a201-9b822bea3687","Type":"ContainerStarted","Data":"4e45cdbd858a3ff5675c2dae4a6c12621fc1d1af7b42f4700eb8aee7b45a9f00"}
Oct 30 12:48:57.831488 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:57.831458    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" event={"ID":"4e6f2bb2-7f4c-4672-a201-9b822bea3687","Type":"ContainerStarted","Data":"17a02fe97965584ab5e181e14177604d3e3e0e0dc6975e2092a8691a1f23f3c8"}
Oct 30 12:48:57.897102 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:57.897034    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podStartSLOduration=1.8970132990000002 podStartE2EDuration="1.897013299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:57.874785896 +0000 UTC m=+781.690112386" watchObservedRunningTime="2026-10-30 12:48:57.897013299 +0000 UTC m=+781.712339792"
Oct 30 12:48:57.922555 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:57.922519    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:48:58.259051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:58.259016    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 12:48:58.259232 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:58.259063    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 12:48:58.316422 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:58.312152    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:58.744862 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:58.744695    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 12:48:58.744862 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:58.744758    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 12:48:58.935385 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:58.934492    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:48:58.935385 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:58.934584    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 12:48:59.259791 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:59.259759    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 12:48:59.259999 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:59.259977    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 12:48:59.732265 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:59.732214    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 12:48:59.732466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:59.732275    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 12:48:59.922472 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:59.922432    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:48:59.923047 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:48:59.923010    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 12:49:00.258531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:00.258495    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 12:49:00.258817 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:00.258790    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 12:49:00.318986 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:00.318903    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:00.737330 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:00.737303    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:49:00.737734 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:00.737714    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:49:00.738415 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:00.738400    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 12:49:00.937121 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:00.937067    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:49:00.939433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:00.939394    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 12:49:01.259037 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:01.259004    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 12:49:01.259213 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:01.259061    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 12:49:01.922830 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:01.922792    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:49:01.923007 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:01.922847    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 12:49:02.138093 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:02.138050    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:02.262433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:02.261726    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 12:49:02.262433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:02.261783    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 12:49:02.922722 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:02.922687    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:49:02.922881 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:02.922750    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 12:49:03.258898 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:03.258818    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 12:49:03.258898 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:03.258873    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 12:49:03.922089 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:03.922055    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:49:03.922299 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:03.922123    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 12:49:03.922788 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:03.922768    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:04.262903 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:04.262819    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:49:04.263412 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:04.263236    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:49:04.263914 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:04.263896    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 12:49:04.926423 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:04.926395    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:49:04.926696 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:04.926677    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:49:04.927261 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:04.927240    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 12:49:05.805397 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:05.802040    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:07.522569 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.522534    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:07.781468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.781391    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"]
Oct 30 12:49:07.786802 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.786776    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.810552 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.810529    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"]
Oct 30 12:49:07.841566 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.841536    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.841679 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.841598    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f949644a-2fa4-48c4-a442-75846a73c62f-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.841740 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.841714    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s4bl\" (UniqueName: \"kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-kube-api-access-2s4bl\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.841890 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.841872    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f949644a-2fa4-48c4-a442-75846a73c62f-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.841936 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.841921    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.842005 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.841992    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.842041 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.842013    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.842041 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.842028    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.842125 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.842067    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.943231 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.943205    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.943362 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.943255    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f949644a-2fa4-48c4-a442-75846a73c62f-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.943415 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.943383    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s4bl\" (UniqueName: \"kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-kube-api-access-2s4bl\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.943537 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.943520    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f949644a-2fa4-48c4-a442-75846a73c62f-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.943613 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.943580    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.943671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.943623    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.943671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.943633    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.943671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.943658    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.943804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.943682    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.943804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.943736    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.943961 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.943937    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.944008 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.943963    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f949644a-2fa4-48c4-a442-75846a73c62f-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.944048 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.943992    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.944294 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.944270    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.945955 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.945937    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.946026 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.945943    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f949644a-2fa4-48c4-a442-75846a73c62f-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.962125 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.962104    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:07.962125 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:07.962112    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s4bl\" (UniqueName: \"kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-kube-api-access-2s4bl\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:08.101540 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:08.101486    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:08.422517 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:49:08.422487    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf949644a_2fa4_48c4_a442_75846a73c62f.slice/crio-3c3326505a137117237709f264af56504d2ef36bee67cb81889c881db4b4079d WatchSource:0}: Error finding container 3c3326505a137117237709f264af56504d2ef36bee67cb81889c881db4b4079d: Status 404 returned error can't find the container with id 3c3326505a137117237709f264af56504d2ef36bee67cb81889c881db4b4079d
Oct 30 12:49:08.423218 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:08.423196    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"]
Oct 30 12:49:08.424807 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:08.424772    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:08.424890 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:08.424834    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:08.424890 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:08.424870    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:08.927885 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:08.927829    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" event={"ID":"f949644a-2fa4-48c4-a442-75846a73c62f","Type":"ContainerStarted","Data":"7806b84e4522665a0e3589c407fe799c9892798d05b2ff7289930a237c29a24d"}
Oct 30 12:49:08.928395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:08.927899    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" event={"ID":"f949644a-2fa4-48c4-a442-75846a73c62f","Type":"ContainerStarted","Data":"3c3326505a137117237709f264af56504d2ef36bee67cb81889c881db4b4079d"}
Oct 30 12:49:09.102300 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:09.102274    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:09.103617 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:09.103588    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:49:09.103751 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:09.103639    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 12:49:09.237938 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:09.237873    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:10.102854 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:10.102818    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:49:10.103289 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:10.102871    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 12:49:11.052509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:11.052469    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:11.115423 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:11.111832    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:49:11.115423 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:11.111927    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 12:49:12.102427 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:12.102389    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:49:12.102592 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:12.102445    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 12:49:12.940118 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:12.940081    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:13.102862 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:13.102817    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:49:13.103063 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:13.102881    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 12:49:14.102000 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:14.101963    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:49:14.102415 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:14.102011    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 12:49:14.827968 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:14.827923    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:15.102627 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:15.102537    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 12:49:15.102627 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:15.102595    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 12:49:16.106270 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:16.106239    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:16.106660 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:16.106570    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:16.107118 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:16.107087    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 12:49:16.162081 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:16.162036    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" podStartSLOduration=9.162024241 podStartE2EDuration="9.162024241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:08.992285605 +0000 UTC m=+792.807612105" watchObservedRunningTime="2026-10-30 12:49:16.162024241 +0000 UTC m=+799.977350729"
Oct 30 12:49:16.554789 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:16.554758    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:17.143348 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:17.143319    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:49:18.363994 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:18.363900    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:19.502558 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:19.502425    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:19.944379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:19.943124    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:19.998734 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:19.998675    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:20.209524 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:20.209425    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:20.362481 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:20.362437    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:49:20.458970 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:20.458905    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:49:20.473505 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:20.473422    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:20.713827 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:20.713790    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:20.945486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:20.945437    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:20.946304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:20.946279    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:49:20.978475 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:20.978442    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:49:20.978727 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:20.978696    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:20.999651 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:20.999608    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:21.000222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:21.000198    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:49:21.145460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:21.145414    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:49:21.200772 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:21.196437    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:21.210512 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:21.210474    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:49:21.211943 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:21.211092    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:49:21.251835 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:21.250788    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:49:21.255392 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:49:21.255132    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:49:21.272057 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:21.272030    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:49:21.493075 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:21.492964    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:49:21.493867 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:21.493844    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:49:21.494086 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:21.494061    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:49:21.602622 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:21.602593    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:49:22.493236 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:22.493201    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:24.116389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:24.116333    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:25.819927 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:25.819900    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:27.606266 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:27.606234    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:30.135376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:30.134480    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:30.267532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:30.266422    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:30.291396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:30.288591    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:30.362469 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:30.362429    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:30.478512 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:30.478418    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:49:30.498374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:30.496536    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:30.606484 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:30.606438    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:30.848365 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:30.848311    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:30.950710 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:30.950657    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:31.239444 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.239049    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:31.239444 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.239249    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:31.244377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.240470    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:49:31.244377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.240911    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:49:31.285433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.275727    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:31.285433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.276177    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:49:31.308627 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.301725    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:31.308627 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.302158    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:49:31.377934 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.377539    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:31.698534 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.693079    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:31.711438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.699225    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:31.900435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.700017    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:31.900435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.700418    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:49:31.900435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.816313    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:49:31.900435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.816377    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:49:31.900435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.816413    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:49:31.900435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.701113    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:31.900435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.876699    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:49:31.900435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:31.876755    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:32.004573 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:32.004492    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:49:32.145385 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:32.131772    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:49:32.156230 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:32.156125    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:49:32.210339 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:32.210314    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:49:32.243489 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:32.243458    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:49:32.254391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:32.251886    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:49:32.310056 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:32.310019    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:49:32.350132 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:32.349841    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:49:32.350132 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:32.349897    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:49:32.357992 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:32.357960    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:49:32.909383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:32.909342    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:34.210921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:34.210876    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:34.750623 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:34.750572    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:36.535762 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:36.535724    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:37.512931 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.512896    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"]
Oct 30 12:49:37.517899 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.517880    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.539907 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.539882    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"]
Oct 30 12:49:37.638003 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.637979    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.638150 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.638013    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.638150 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.638034    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.638238 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.638154    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.638238 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.638188    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.638238 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.638217    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.638444 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.638328    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwt6q\" (UniqueName: \"kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-kube-api-access-bwt6q\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.638444 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.638420    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.638525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.638456    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.739998 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.739973    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.740121 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.740021    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.740121 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.740040    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.740121 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.740074    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.740121 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.740096    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.740331 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.740125    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.740331 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.740151    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwt6q\" (UniqueName: \"kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-kube-api-access-bwt6q\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.740331 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.740179    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.740331 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.740207    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.740579 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.740535    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.740579 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.740564    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-data\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.740694 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.740662    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-certs\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.740758 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.740738    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-credential-socket\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.740881 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.740860    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istiod-ca-cert\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.742458 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.742440    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-envoy\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.742684 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.742666    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-podinfo\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.767994 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.767944    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwt6q\" (UniqueName: \"kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-kube-api-access-bwt6q\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.768994 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.768970    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-token\") pod \"e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") " pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:37.831995 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:37.831964    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:38.147341 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:49:38.147317    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b94e06d_f102_43e5_86f2_ab0c2065b4ca.slice/crio-9a6c57a9794bc3838e256d7c7322a1cc2f0a79f37768e35f8cd5374068a259a8 WatchSource:0}: Error finding container 9a6c57a9794bc3838e256d7c7322a1cc2f0a79f37768e35f8cd5374068a259a8: Status 404 returned error can't find the container with id 9a6c57a9794bc3838e256d7c7322a1cc2f0a79f37768e35f8cd5374068a259a8
Oct 30 12:49:38.147777 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:38.147756    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"]
Oct 30 12:49:38.149456 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:38.149427    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:38.149527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:38.149485    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:38.149527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:38.149512    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:38.290690 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:38.290663    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:39.016013 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:39.015969    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" event={"ID":"8b94e06d-f102-43e5-86f2-ab0c2065b4ca","Type":"ContainerStarted","Data":"d9a9f205b0574c462664d0ee7330b56607a3d7383f1cc945538c1a7beffdae68"}
Oct 30 12:49:39.016530 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:39.016065    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" event={"ID":"8b94e06d-f102-43e5-86f2-ab0c2065b4ca","Type":"ContainerStarted","Data":"9a6c57a9794bc3838e256d7c7322a1cc2f0a79f37768e35f8cd5374068a259a8"}
Oct 30 12:49:39.093260 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:39.093203    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" podStartSLOduration=2.093189497 podStartE2EDuration="2.093189497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:39.092674025 +0000 UTC m=+822.908000515" watchObservedRunningTime="2026-10-30 12:49:39.093189497 +0000 UTC m=+822.908515983"
Oct 30 12:49:39.832795 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:39.832759    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:39.834159 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:39.834134    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 12:49:39.834289 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:39.834189    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 12:49:40.055568 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:40.055536    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:40.833288 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:40.833256    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 12:49:40.833479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:40.833305    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 12:49:41.700523 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.700490    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"]
Oct 30 12:49:41.704676 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.704654    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.714805 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.714780    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.714951 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.714815    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxq6d\" (UniqueName: \"kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-kube-api-access-dxq6d\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.714951 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.714849    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.714951 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.714880    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.715124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.714958    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.715124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.714992    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.715124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.715027    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.715124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.715056    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.715124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.715099    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.739516 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.739490    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"]
Oct 30 12:49:41.816674 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.816641    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.816855 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.816788    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.816855 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.816806    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxq6d\" (UniqueName: \"kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-kube-api-access-dxq6d\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.816855 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.816826    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.816855 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.816848    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.817074 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.816887    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.817074 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.816918    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.817074 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.816951    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.817074 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.816979    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.817330 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.817298    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-certs\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.817425 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.817331    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-data\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.817494 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.817439    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.817494 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.817454    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-credential-socket\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.817713 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.817695    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istiod-ca-cert\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.819304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.819280    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-envoy\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.819465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.819446    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-podinfo\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.829210 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.829174    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:41.833265 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.833246    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 12:49:41.833361 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.833284    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 12:49:41.835847 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.835819    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-token\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:41.836222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:41.836200    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxq6d\" (UniqueName: \"kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-kube-api-access-dxq6d\") pod \"e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") " pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:42.019144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:42.019120    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:42.355127 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:49:42.355095    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfeffa41_ac7c_4809_a0c0_fa9b2e9412ef.slice/crio-90c05e637c4a87dcbe36afb5c4b84e248cbd0003bff10d251a4fce34b86db41a WatchSource:0}: Error finding container 90c05e637c4a87dcbe36afb5c4b84e248cbd0003bff10d251a4fce34b86db41a: Status 404 returned error can't find the container with id 90c05e637c4a87dcbe36afb5c4b84e248cbd0003bff10d251a4fce34b86db41a
Oct 30 12:49:42.355786 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:42.355711    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"]
Oct 30 12:49:42.357402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:42.357339    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:42.357512 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:42.357445    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:42.357512 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:42.357491    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:42.832907 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:42.832875    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 12:49:42.833370 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:42.832946    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 12:49:43.039758 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:43.039721    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" event={"ID":"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef","Type":"ContainerStarted","Data":"47f88fd4d522d009d64c4a880d593e7578975e00ed156b3e934a294a32a4a3a9"}
Oct 30 12:49:43.039962 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:43.039771    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" event={"ID":"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef","Type":"ContainerStarted","Data":"90c05e637c4a87dcbe36afb5c4b84e248cbd0003bff10d251a4fce34b86db41a"}
Oct 30 12:49:43.688377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:43.686893    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:43.835378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:43.834450    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 12:49:43.835378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:43.834524    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 12:49:44.020601 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:44.020574    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:44.021685 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:44.021662    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 12:49:44.021854 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:44.021835    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 12:49:45.019692 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:45.019647    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 12:49:45.020196 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:45.019708    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 12:49:45.126420 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:45.126390    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:45.126703 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:45.126682    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:45.127495 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:45.127460    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 12:49:45.192759 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:45.192701    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" podStartSLOduration=4.192687454 podStartE2EDuration="4.192687454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:43.104743045 +0000 UTC m=+826.920069533" watchObservedRunningTime="2026-10-30 12:49:45.192687454 +0000 UTC m=+829.008013940"
Oct 30 12:49:45.622167 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:45.622134    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:46.019823 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.019786    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 12:49:46.020245 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.019850    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 12:49:46.347913 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.347834    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"]
Oct 30 12:49:46.351784 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.351761    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.401972 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.401857    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.402285 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.402246    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbjgc\" (UniqueName: \"kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-kube-api-access-cbjgc\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.402468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.402305    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.402468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.402331    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.402592 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.402468    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.402592 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.402516    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.402592 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.402548    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.402754 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.402601    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.402867 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.402819    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.408224 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.408198    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"]
Oct 30 12:49:46.504225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.504196    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.504407 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.504286    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.504407 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.504325    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.504407 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.504401    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbjgc\" (UniqueName: \"kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-kube-api-access-cbjgc\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.504560 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.504429    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.504560 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.504446    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.504560 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.504470    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.504560 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.504497    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.504560 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.504525    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.504828 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.504666    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-data\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.504905 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.504883    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.504964 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.504922    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-certs\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.504964 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.504945    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-credential-socket\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.505030 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.505010    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istiod-ca-cert\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.506906 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.506877    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-envoy\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.507224 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.507205    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-podinfo\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.522509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.522483    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-token\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.522700 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.522683    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbjgc\" (UniqueName: \"kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-kube-api-access-cbjgc\") pod \"e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") " pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:46.701462 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:46.701387    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:47.017639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:47.017568    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"]
Oct 30 12:49:47.019863 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:47.019844    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 12:49:47.020163 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:47.019890    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 12:49:47.029988 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:49:47.029946    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4b7a1f_8e69_4885_86ca_755ac19d97ec.slice/crio-6344a427a94a5b0bccb010ec6714927d3a43f49b11d588f25e9754167b2dbb45 WatchSource:0}: Error finding container 6344a427a94a5b0bccb010ec6714927d3a43f49b11d588f25e9754167b2dbb45: Status 404 returned error can't find the container with id 6344a427a94a5b0bccb010ec6714927d3a43f49b11d588f25e9754167b2dbb45
Oct 30 12:49:47.033759 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:47.033724    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:47.033862 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:47.033794    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:47.033862 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:47.033824    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:47.097778 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:47.097748    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" event={"ID":"fe4b7a1f-8e69-4885-86ca-755ac19d97ec","Type":"ContainerStarted","Data":"6344a427a94a5b0bccb010ec6714927d3a43f49b11d588f25e9754167b2dbb45"}
Oct 30 12:49:47.424848 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:47.424808    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:47.621370 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:47.621329    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:48.019640 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:48.019596    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 12:49:48.019856 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:48.019655    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 12:49:48.121106 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:48.121060    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" event={"ID":"fe4b7a1f-8e69-4885-86ca-755ac19d97ec","Type":"ContainerStarted","Data":"028b8a9800f563ceeecf69d718d096218d792868f7afe945d77a03198d9fc1d9"}
Oct 30 12:49:48.701953 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:48.701903    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:48.703671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:48.703644    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 12:49:48.703808 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:48.703695    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 12:49:49.344853 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:49.344766    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:49.345322 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:49.345298    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:49.346512 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:49.346469    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 12:49:49.403361 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:49.403308    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" podStartSLOduration=3.403295697 podStartE2EDuration="3.403295697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:48.191644559 +0000 UTC m=+832.006971049" watchObservedRunningTime="2026-10-30 12:49:49.403295697 +0000 UTC m=+833.218622185"
Oct 30 12:49:49.701808 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:49.701716    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 12:49:49.701808 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:49.701776    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 12:49:49.791377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:49.789689    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:50.702585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:50.702546    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 12:49:50.703075 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:50.702602    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 12:49:51.132648 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:49:51.132613    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:49:51.702121 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:51.702085    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 12:49:51.702313 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:51.702164    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 12:49:51.890374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:51.890117    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:52.701968 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:52.701933    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 12:49:52.702144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:52.701996    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 12:49:52.918002 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:52.917966    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"]
Oct 30 12:49:52.921967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:52.921949    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:52.964392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:52.963136    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"]
Oct 30 12:49:53.004878 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.004845    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.005116 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.004913    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.005116 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.004990    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.005116 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.005018    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6384f31c-d523-4d6d-ad87-e531be8bd74a-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.005116 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.005062    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.005346 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.005124    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.005346 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.005237    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.005346 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.005308    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpdpf\" (UniqueName: \"kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-kube-api-access-vpdpf\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.005529 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.005367    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.106147 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.106110    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpdpf\" (UniqueName: \"kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-kube-api-access-vpdpf\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.106314 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.106157    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.106314 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.106180    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.106314 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.106207    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.106314 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.106248    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.106314 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.106262    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6384f31c-d523-4d6d-ad87-e531be8bd74a-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.106314 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.106291    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.106663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.106346    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.106663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.106472    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.106663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.106638    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-certs\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.106783 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.106674    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-credential-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.106783 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.106751    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-socket\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.109246 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.107324    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-data\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.109246 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.107486    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6384f31c-d523-4d6d-ad87-e531be8bd74a-istiod-ca-cert\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.109648 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.109616    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-envoy\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.112281 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.110219    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-podinfo\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.124477 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.124455    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-token\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.124942 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.124918    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpdpf\" (UniqueName: \"kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-kube-api-access-vpdpf\") pod \"e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") " pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.237540 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.237470    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:53.560847 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:49:53.560818    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6384f31c_d523_4d6d_ad87_e531be8bd74a.slice/crio-bf657b6fde0eafa96b6ea7c38412da6573646c4d0c51aa38c78c8b56a9f5eebd WatchSource:0}: Error finding container bf657b6fde0eafa96b6ea7c38412da6573646c4d0c51aa38c78c8b56a9f5eebd: Status 404 returned error can't find the container with id bf657b6fde0eafa96b6ea7c38412da6573646c4d0c51aa38c78c8b56a9f5eebd
Oct 30 12:49:53.561495 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.561468    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"]
Oct 30 12:49:53.563461 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.563419    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:53.563551 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.563532    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:53.563602 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.563580    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:53.701835 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.701798    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 12:49:53.701985 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.701861    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 12:49:53.880241 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:53.880142    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:54.159287 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:54.159186    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" event={"ID":"6384f31c-d523-4d6d-ad87-e531be8bd74a","Type":"ContainerStarted","Data":"7ae07040c0e0351adbd11052029b1c50dbed9f1d7a1bacacba1f4d58d1909cfd"}
Oct 30 12:49:54.159287 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:54.159258    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" event={"ID":"6384f31c-d523-4d6d-ad87-e531be8bd74a","Type":"ContainerStarted","Data":"bf657b6fde0eafa96b6ea7c38412da6573646c4d0c51aa38c78c8b56a9f5eebd"}
Oct 30 12:49:54.203344 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:54.203288    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" podStartSLOduration=2.203274778 podStartE2EDuration="2.203274778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:54.200952948 +0000 UTC m=+838.016279489" watchObservedRunningTime="2026-10-30 12:49:54.203274778 +0000 UTC m=+838.018601315"
Oct 30 12:49:54.239844 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:54.239809    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:49:54.241848 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:54.241295    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 12:49:54.241848 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:54.241389    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 12:49:54.706514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:54.706482    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:54.706794 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:54.706765    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:54.707522 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:54.707503    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 12:49:55.238438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:55.238407    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 12:49:55.238814 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:55.238453    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 12:49:55.720489 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:55.720431    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:56.110186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.110142    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"]
Oct 30 12:49:56.115108 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.115084    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.124532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.124513    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"]
Oct 30 12:49:56.128127 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.128109    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.138532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.137111    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"]
Oct 30 12:49:56.154050 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.154028    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"]
Oct 30 12:49:56.154684 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.154655    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.154797 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.154748    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.154797 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.154773    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2019baf-7b8e-4c05-96b8-b48f93809985-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.154936 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.154802    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.154936 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.154839    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.154936 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.154901    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.155113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.155008    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.155113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.155087    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.155229 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.155167    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.155229 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.155204    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.155334 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.155228    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.155334 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.155252    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.155334 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.155288    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.155334 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.155319    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qnv\" (UniqueName: \"kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-kube-api-access-w7qnv\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.155587 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.155344    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nctz2\" (UniqueName: \"kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-kube-api-access-nctz2\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.155587 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.155402    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.155587 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.155427    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.155587 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.155464    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.238577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.238548    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 12:49:56.239072 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.238593    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 12:49:56.256900 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.256869    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.257130 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257105    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2019baf-7b8e-4c05-96b8-b48f93809985-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.257225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257154    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.257225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257216    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.257396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257288    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.257471 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257429    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.257548 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257533    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.257628 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257614    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.257684 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257665    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.257742 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257696    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.257742 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257703    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.257742 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257724    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.257742 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257729    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.257934 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257800    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.257934 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257847    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qnv\" (UniqueName: \"kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-kube-api-access-w7qnv\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.257934 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257880    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nctz2\" (UniqueName: \"kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-kube-api-access-nctz2\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.258083 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257943    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.258083 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257954    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.258083 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.257974    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.258083 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.258016    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.258083 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.258066    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.258328 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.258110    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-certs\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.258815 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.258792    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2019baf-7b8e-4c05-96b8-b48f93809985-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.259098 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.259076    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.259390 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.259344    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istiod-ca-cert\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.259485 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.259396    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.259661 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.259636    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-credential-socket\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.261487 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.260544    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.261487 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.260792    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-data\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.262482 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.262462    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-envoy\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.263470 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.263446    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.264651 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.264631    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-podinfo\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.305255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.302404    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.305255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.302755    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nctz2\" (UniqueName: \"kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-kube-api-access-nctz2\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.305255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.302991    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-token\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.305255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.305031    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qnv\" (UniqueName: \"kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-kube-api-access-w7qnv\") pod \"e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") " pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.430406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.430312    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:56.444106 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.444080    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:56.794375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.793962    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"]
Oct 30 12:49:56.798426 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:49:56.798402    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6aa45d0_5cee_463c_a2d4_89ee83f50aa1.slice/crio-313d678e259ecdda19f984a9026fab2695f2b1fb79cb46ac43ff4080458d2a14 WatchSource:0}: Error finding container 313d678e259ecdda19f984a9026fab2695f2b1fb79cb46ac43ff4080458d2a14: Status 404 returned error can't find the container with id 313d678e259ecdda19f984a9026fab2695f2b1fb79cb46ac43ff4080458d2a14
Oct 30 12:49:56.800725 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.800694    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:56.800783 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.800758    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:56.800830 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.800785    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:56.815308 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:49:56.815282    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2019baf_7b8e_4c05_96b8_b48f93809985.slice/crio-aee61c8072e2631f897ea9a3e0e370731cf1b7a45ba6095ba4efab8c01cbfc2f WatchSource:0}: Error finding container aee61c8072e2631f897ea9a3e0e370731cf1b7a45ba6095ba4efab8c01cbfc2f: Status 404 returned error can't find the container with id aee61c8072e2631f897ea9a3e0e370731cf1b7a45ba6095ba4efab8c01cbfc2f
Oct 30 12:49:56.817885 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.817845    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:56.817977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.817933    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:56.817977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.817965    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"]
Oct 30 12:49:56.818089 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:56.817975    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:49:57.215753 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.215659    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" event={"ID":"f2019baf-7b8e-4c05-96b8-b48f93809985","Type":"ContainerStarted","Data":"acd717be8d37c52f00bd759a11b599d464c9623603dbde4fe1780e41b0ab1b37"}
Oct 30 12:49:57.215753 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.215725    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" event={"ID":"f2019baf-7b8e-4c05-96b8-b48f93809985","Type":"ContainerStarted","Data":"aee61c8072e2631f897ea9a3e0e370731cf1b7a45ba6095ba4efab8c01cbfc2f"}
Oct 30 12:49:57.219328 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.219299    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" event={"ID":"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1","Type":"ContainerStarted","Data":"3831989687ff66bde49bde3408b4c3c0a80cc8d1f4329273e8ff6f5c44b351f8"}
Oct 30 12:49:57.219455 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.219365    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" event={"ID":"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1","Type":"ContainerStarted","Data":"313d678e259ecdda19f984a9026fab2695f2b1fb79cb46ac43ff4080458d2a14"}
Oct 30 12:49:57.237937 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.237908    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 12:49:57.238088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.237954    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 12:49:57.260076 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.260032    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" podStartSLOduration=1.260018947 podStartE2EDuration="1.260018947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:57.258505056 +0000 UTC m=+841.073831544" watchObservedRunningTime="2026-10-30 12:49:57.260018947 +0000 UTC m=+841.075345494"
Oct 30 12:49:57.430975 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.430876    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:49:57.432913 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.432879    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 12:49:57.433048 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.432937    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 12:49:57.445209 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.445183    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:49:57.446641 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.446615    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 12:49:57.446786 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.446667    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 12:49:57.529390 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:57.526760    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:58.245992 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:58.245954    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 12:49:58.246274 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:58.246242    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 12:49:58.441128 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:58.441087    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 12:49:58.443521 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:58.443465    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 12:49:58.445170 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:58.445078    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 12:49:58.445170 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:58.445131    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 12:49:59.254803 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:59.254757    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 12:49:59.255096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:59.255060    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 12:49:59.434376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:59.434037    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 12:49:59.434376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:59.434098    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 12:49:59.452462 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:59.450845    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 12:49:59.452462 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:59.450923    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 12:49:59.512489 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:49:59.512390    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:00.238741 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:00.238697    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 12:50:00.238970 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:00.238756    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 12:50:00.437564 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:00.432724    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 12:50:00.437564 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:00.432798    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 12:50:00.444582 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:00.444555    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 12:50:00.444718 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:00.444600    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 12:50:01.242998 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:01.242714    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:50:01.243685 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:01.243552    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:50:01.244651 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:01.244632    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 12:50:01.326410 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:01.322247    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" podStartSLOduration=5.32222682 podStartE2EDuration="5.32222682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:57.31268187 +0000 UTC m=+841.128008357" watchObservedRunningTime="2026-10-30 12:50:01.32222682 +0000 UTC m=+845.137553308"
Oct 30 12:50:01.336994 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:01.336963    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:01.442343 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:01.441183    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 12:50:01.442343 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:01.441226    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 12:50:01.444803 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:01.444706    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 12:50:01.444803 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:01.444772    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 12:50:01.516429 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:01.504882    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:02.431072 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:02.431032    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 12:50:02.431497 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:02.431093    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 12:50:02.446077 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:02.446055    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 12:50:02.446250 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:02.446234    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 12:50:03.120816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:03.120780    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:50:03.427924 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:03.427822    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:03.430986 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:03.430938    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 12:50:03.431127 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:03.431005    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 12:50:03.445334 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:03.445312    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 12:50:03.445455 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:03.445370    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 12:50:05.166419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.162216    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:05.170802 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.167060    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:05.182297 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.182266    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:50:05.183733 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.183713    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:50:05.230664 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.230629    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 12:50:05.264380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.250801    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:05.331502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.331448    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:05.491517 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.488246    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:50:05.491767 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.491735    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:05.596677 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.595675    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:50:05.596677 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.596618    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:50:05.613603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.613529    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 12:50:05.761428 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.758031    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:05.990853 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:05.989386    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:06.190781 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:06.190736    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:06.193219 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:06.193191    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:50:06.235991 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:06.233438    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:06.236734 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:06.236704    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:06.238280 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:06.237439    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:50:06.443686 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:06.375005    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:50:06.443686 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:06.376319    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:50:06.443686 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:06.415256    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:50:07.039501 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.039443    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:07.250762 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.113476    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:50:07.280473 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.280423    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:07.288469 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.242499    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:50:07.317435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.242461    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:07.340763 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.304762    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:50:07.341096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.341073    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:50:07.344463 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.344440    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:50:07.344639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.344625    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:50:07.349371 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.307042    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:07.380453 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.376082    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:50:07.381115 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.381085    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:50:07.447377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.412730    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:50:07.447377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.444434    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:07.982379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.972778    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:08.007377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:07.986701    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:08.257610 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:08.257513    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:08.296229 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:08.296185    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:08.344261 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:08.344219    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:08.395291 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:08.395240    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:08.439961 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:08.439917    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:08.475484 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:08.440972    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:08.495545 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:08.465815    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:50:09.316375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:09.302061    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:50:09.319496 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:09.319456    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:50:09.348925 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:09.348484    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:09.397510 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:09.397461    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:09.496585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:09.479790    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:50:09.581428 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:09.506456    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:09.581428 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:09.508447    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:09.581428 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:09.533433    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:50:10.068699 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:10.067602    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:50:10.096573 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:10.095757    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:50:10.227754 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:10.226989    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:50:10.298575 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:10.298547    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:50:10.312639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:10.311756    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:50:10.315862 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:10.315834    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:50:10.318177 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:10.318153    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:50:10.378514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:10.378472    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:12.279953 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:12.279918    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:14.146885 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:14.146849    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:16.043070 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:16.043038    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:17.726318 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:17.726284    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:19.407203 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:19.407167    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:20.897838 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:20.897805    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"]
Oct 30 12:50:20.902507 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:20.902479    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:20.922130 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:20.922102    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"]
Oct 30 12:50:20.928809 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:20.928789    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4cd3fb\"/\"default-dockercfg-slm6x\""
Oct 30 12:50:20.929178 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:20.929160    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4cd3fb\"/\"kube-root-ca.crt\""
Oct 30 12:50:20.929259 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:20.929183    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4cd3fb\"/\"openshift-service-ca.crt\""
Oct 30 12:50:20.947845 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:20.947817    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:21.017885 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.017858    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4cd3fb\"/\"distinct-w0-4cd3fb-kserve-self-signed-certs\""
Oct 30 12:50:21.047704 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.047678    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:21.078026 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.078000    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-home\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.078105 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.078045    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sb2b\" (UniqueName: \"kubernetes.io/projected/91ef2bf6-d815-459d-8162-2dc297e01897-kube-api-access-9sb2b\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.078169 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.078156    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-model-cache\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.078210 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.078191    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-dshm\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.078276 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.078266    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-tmp-dir\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.078311 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.078286    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef2bf6-d815-459d-8162-2dc297e01897-tls-certs\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.184272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.183247    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-tmp-dir\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.184272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.183295    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef2bf6-d815-459d-8162-2dc297e01897-tls-certs\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.184272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.183432    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-home\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.184272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.183487    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sb2b\" (UniqueName: \"kubernetes.io/projected/91ef2bf6-d815-459d-8162-2dc297e01897-kube-api-access-9sb2b\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.184272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.183612    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-model-cache\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.184272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.183664    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-dshm\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.186010 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.185081    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-tmp-dir\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.192415 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.186592    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-home\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.192651 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.190033    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-model-cache\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.214190 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.214111    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef2bf6-d815-459d-8162-2dc297e01897-tls-certs\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.215187 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.215147    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-dshm\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.256374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.250250    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sb2b\" (UniqueName: \"kubernetes.io/projected/91ef2bf6-d815-459d-8162-2dc297e01897-kube-api-access-9sb2b\") pod \"distinct-w0-4cd3fb-kserve-76799d77df-k5tpw\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") " pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.285714 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:50:21.285676    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:50:21.526259 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.526221    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:21.973436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:21.948329    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"]
Oct 30 12:50:22.552787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:22.552755    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" event={"ID":"91ef2bf6-d815-459d-8162-2dc297e01897","Type":"ContainerStarted","Data":"20faa40ab9e6eb37d16905a0eda79784f0bad2ad8ca0fdabe3a56b2e2ce3fcbc"}
Oct 30 12:50:22.552972 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:22.552795    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" event={"ID":"91ef2bf6-d815-459d-8162-2dc297e01897","Type":"ContainerStarted","Data":"5453de70d9c33026da98ee10008a882e4573f0a026a6e5364ac41ad02bf7d7d8"}
Oct 30 12:50:22.593497 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:22.593452    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podStartSLOduration=2.5934396570000002 podStartE2EDuration="2.593439657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:22.590428579 +0000 UTC m=+866.405755068" watchObservedRunningTime="2026-10-30 12:50:22.593439657 +0000 UTC m=+866.408766144"
Oct 30 12:50:22.902191 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:22.902115    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:23.526666 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:23.526636    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:24.701585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:24.701555    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:26.542674 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:26.542641    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:27.101188 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:27.101160    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:27.200477 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:27.200447    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:28.220336 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:28.220306    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:29.908913 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:29.908853    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:30.220849 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:30.220770    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:31.526789 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:31.526755    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:31.540421 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:31.540397    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:31.613828 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:31.613804    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:31.715218 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:31.715188    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:32.334452 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:32.334426    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:50:33.604663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:33.604620    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:35.318584 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:35.318449    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:35.320484 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:35.320442    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:35.437897 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:35.437860    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:50:35.438210 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:35.438176    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:35.512068 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:35.512023    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:50:35.651987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:35.651892    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:36.017801 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:36.017694    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:50:36.018123 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:36.018092    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:36.324547 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:36.324494    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:50:36.326065 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:36.326046    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:50:36.327871 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:36.327847    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:50:36.336388 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:36.328395    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:50:36.353218 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:36.350758    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:50:36.452320 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:36.404187    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:36.452320 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:36.404557    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:36.509966 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:36.485759    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:36.719433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:36.716689    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:36.743536 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:36.740537    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:37.029437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.024725    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:50:37.029437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.024788    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:37.339113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.338992    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:50:37.352513 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.352427    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:37.381391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.362667    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:50:37.457505 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.457426    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:50:37.458703 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.385108    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:37.458871 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.422641    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:50:37.460815 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.422609    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:50:37.462023 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.462004    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:50:37.462171 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.462158    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:50:37.508848 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.503467    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:37.508848 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.505458    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:50:37.723941 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.723838    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:37.798390 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:37.763459    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:50:38.350535 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:38.350485    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:38.360479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:38.360435    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:38.482260 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:38.482217    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:38.482792 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:38.482760    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:50:38.484569 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:38.484531    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:50:38.509383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:38.504800    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:50:38.726491 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:38.724459    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:50:38.773377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:38.766786    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:50:38.792105 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:38.787841    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:50:39.266423 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:39.252048    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:39.352681 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:39.352637    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:39.497387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:39.488279    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:50:39.498093 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:39.498054    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:50:39.506902 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:39.501650    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:39.522641 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:39.501681    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:50:39.523421 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:39.506447    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:50:39.733626 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:39.733585    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:50:39.925663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:39.925562    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:50:40.027654 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.027620    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:50:40.088416 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.087489    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"]
Oct 30 12:50:40.099878 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.093588    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"]
Oct 30 12:50:40.100090 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.094454    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:40.126721 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.126695    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dddj8\""
Oct 30 12:50:40.127009 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.126990    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 12:50:40.127211 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.127195    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 12:50:40.127438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.127421    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 12:50:40.127628 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.127612    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f9689ecfae68\""
Oct 30 12:50:40.127740 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.127712    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 12:50:40.301660 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.301627    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-plugins-config-volume\") pod \"payload-processing-external-model-79d5cf7c65-qm5df\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:40.302429 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.302402    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czcld\" (UniqueName: \"kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-kube-api-access-czcld\") pod \"payload-processing-external-model-79d5cf7c65-qm5df\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:40.302730 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.302714    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-routing-overlay-volume\") pod \"payload-processing-external-model-79d5cf7c65-qm5df\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:40.302934 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.302908    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-provider-credentials\") pod \"payload-processing-external-model-79d5cf7c65-qm5df\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:40.382595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.382557    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:40.391639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.386519    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:50:40.411372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.408114    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-routing-overlay-volume\") pod \"payload-processing-external-model-79d5cf7c65-qm5df\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:40.411372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.408285    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-provider-credentials\") pod \"payload-processing-external-model-79d5cf7c65-qm5df\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:40.411372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.408490    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-plugins-config-volume\") pod \"payload-processing-external-model-79d5cf7c65-qm5df\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:40.411372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.408518    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czcld\" (UniqueName: \"kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-kube-api-access-czcld\") pod \"payload-processing-external-model-79d5cf7c65-qm5df\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:40.411372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.409662    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-routing-overlay-volume\") pod \"payload-processing-external-model-79d5cf7c65-qm5df\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:40.412801 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.412777    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-plugins-config-volume\") pod \"payload-processing-external-model-79d5cf7c65-qm5df\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:40.428749 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.428720    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-provider-credentials\") pod \"payload-processing-external-model-79d5cf7c65-qm5df\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:40.437082 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.437055    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:50:40.437254 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.437110    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:50:40.437254 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.437138    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:50:40.437254 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.437163    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:50:40.439344 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.438785    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:50:40.441550 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.441514    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czcld\" (UniqueName: \"kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-kube-api-access-czcld\") pod \"payload-processing-external-model-79d5cf7c65-qm5df\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") " pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:40.724552 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:40.724474    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:41.031995 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:50:41.031969    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9419a58f_4844_4c6c_a208_eb64b2785447.slice/crio-9aa04d321a8c64c2b1ae9479b3edc21d59a934d64e9f5b4cb23fa3334df9d448 WatchSource:0}: Error finding container 9aa04d321a8c64c2b1ae9479b3edc21d59a934d64e9f5b4cb23fa3334df9d448: Status 404 returned error can't find the container with id 9aa04d321a8c64c2b1ae9479b3edc21d59a934d64e9f5b4cb23fa3334df9d448
Oct 30 12:50:41.034051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:41.034022    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"]
Oct 30 12:50:41.324298 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:41.324226    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:41.569202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:41.569164    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df" event={"ID":"9419a58f-4844-4c6c-a208-eb64b2785447","Type":"ContainerStarted","Data":"c9b90cb835ae0608be10d6a34c0e7ce390f8cd527d21bc832ac4d640b4c0e13e"}
Oct 30 12:50:41.569617 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:41.569223    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df" event={"ID":"9419a58f-4844-4c6c-a208-eb64b2785447","Type":"ContainerStarted","Data":"9aa04d321a8c64c2b1ae9479b3edc21d59a934d64e9f5b4cb23fa3334df9d448"}
Oct 30 12:50:41.569617 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:41.569253    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:41.608413 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:41.608323    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df" podStartSLOduration=2.608311352 podStartE2EDuration="2.608311352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:41.607435922 +0000 UTC m=+885.422762411" watchObservedRunningTime="2026-10-30 12:50:41.608311352 +0000 UTC m=+885.423637838"
Oct 30 12:50:43.025345 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:43.025315    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:44.812341 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:44.812300    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:46.599394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:46.599335    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:47.554936 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:47.554896    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:48.305189 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:48.305154    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:49.910438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:49.910404    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:51.133371 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:50:51.133327    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:50:51.624869 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:51.624820    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:51.723987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:51.723957    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:52.577645 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:52.577618    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:50:53.518602 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:53.518558    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:55.361188 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:55.361159    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:56.996444 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:56.996413    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:58.524035 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:58.523992    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:50:58.706121 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:50:58.706091    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:01.188375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:01.174103    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:01.357505 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:01.357454    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:01.403538 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:01.403489    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:01.436378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:01.422482    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:01.507619 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:01.506425    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:01.511967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:01.511931    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:01.512456 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:01.512428    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded"
Oct 30 12:51:01.545422 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:01.545387    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:51:01.626783 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:01.626747    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:01.744817 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:01.742452    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:51:01.775119 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:01.775076    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:01.793595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:01.793463    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:01.793595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:01.793530    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:02.214162 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.214074    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:02.421380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.396896    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:51:02.421380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.399023    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:02.421380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.399069    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:02.445416 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.424270    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:02.492974 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.447742    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:02.521377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.448846    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:51:02.527513 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.527488    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:51:02.527757 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.527737    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:51:02.531387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.525677    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:51:02.531387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.530837    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:51:02.535368 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.525721    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:02.539733 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.539712    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:51:02.560377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.548423    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:51:02.560377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.548765    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded"
Oct 30 12:51:02.560612 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.560476    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:51:02.743429 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.743303    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:02.816374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.812535    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:51:02.831376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.829025    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:51:02.831376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:02.829095    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:51:03.460377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:03.459623    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:51:03.550991 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:03.498620    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:03.550991 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:03.500197    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:03.550991 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:03.533435    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:03.550991 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:03.544117    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:51:03.568395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:03.568321    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:51:03.771441 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:03.747646    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:03.816378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:03.812872    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:03.859663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:03.859625    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:51:03.916653 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:03.916623    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:51:04.317376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:04.303000    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:04.460759 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:04.460716    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:04.509687 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:04.509645    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:04.510763 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:04.510015    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:04.668432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:04.666480    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:51:04.677375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:04.673420    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:04.700401 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:04.697956    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:05.043509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.043461    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:05.357697 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.357611    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:51:05.428378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.420459    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:05.428378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.420526    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:05.512377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.511519    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:05.514454 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.514418    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:05.672227 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.671495    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:05.694644 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.692652    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:51:05.846543 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.795452    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:05.884382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.865198    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:05.985834 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.985431    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:05.985834 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.985491    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:05.989076 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.989050    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:51:05.989298 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:05.989282    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:51:06.058199 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:06.058169    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:51:06.334581 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:06.334547    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:06.611998 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:06.611886    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:51:06.692837 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:06.692795    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:06.747587 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:06.747551    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:51:06.821669 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:06.821640    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:51:07.096765 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:07.096730    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:08.215490 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:08.215451    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:09.653529 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:09.653493    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:09.742857 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:09.742822    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:09.986424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:09.984903    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:11.285144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.285110    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"]
Oct 30 12:51:11.290655 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.290633    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.308019 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.307994    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"]
Oct 30 12:51:11.315953 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.315930    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4331a0c49658\""
Oct 30 12:51:11.316066 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.315933    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-515b6b28555f\""
Oct 30 12:51:11.416067 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.416033    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-plugins-config-volume\") pod \"payload-processing-external-model-7d97d47b88-5l9wk\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") " pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.416217 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.416099    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-routing-overlay-volume\") pod \"payload-processing-external-model-7d97d47b88-5l9wk\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") " pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.416291 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.416263    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp84w\" (UniqueName: \"kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-kube-api-access-zp84w\") pod \"payload-processing-external-model-7d97d47b88-5l9wk\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") " pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.416365 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.416309    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-provider-credentials\") pod \"payload-processing-external-model-7d97d47b88-5l9wk\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") " pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.517480 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.517456    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-plugins-config-volume\") pod \"payload-processing-external-model-7d97d47b88-5l9wk\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") " pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.517605 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.517507    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-routing-overlay-volume\") pod \"payload-processing-external-model-7d97d47b88-5l9wk\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") " pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.517605 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.517573    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp84w\" (UniqueName: \"kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-kube-api-access-zp84w\") pod \"payload-processing-external-model-7d97d47b88-5l9wk\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") " pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.517716 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.517607    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-provider-credentials\") pod \"payload-processing-external-model-7d97d47b88-5l9wk\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") " pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.518146 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.518116    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-routing-overlay-volume\") pod \"payload-processing-external-model-7d97d47b88-5l9wk\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") " pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.518257 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.518202    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-plugins-config-volume\") pod \"payload-processing-external-model-7d97d47b88-5l9wk\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") " pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.520169 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.520142    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-provider-credentials\") pod \"payload-processing-external-model-7d97d47b88-5l9wk\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") " pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.535793 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.535736    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp84w\" (UniqueName: \"kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-kube-api-access-zp84w\") pod \"payload-processing-external-model-7d97d47b88-5l9wk\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") " pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.603649 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.603623    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:11.905616 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.905537    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"]
Oct 30 12:51:11.921107 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.921076    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk" event={"ID":"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b","Type":"ContainerStarted","Data":"6550e3e6e9a373412a8eda029b96b6bd5d077dc600262f865f54a18615bf2fae"}
Oct 30 12:51:11.926933 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:11.926903    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:12.929841 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:12.929803    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk" event={"ID":"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b","Type":"ContainerStarted","Data":"c52d5537ca922f0f0a40559aa9499d552486601d7d5c5cbf6c12c5c0b6d8538b"}
Oct 30 12:51:12.930327 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:12.929860    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:12.970007 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:12.969952    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk" podStartSLOduration=1.969940107 podStartE2EDuration="1.969940107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:12.967075304 +0000 UTC m=+916.782401792" watchObservedRunningTime="2026-10-30 12:51:12.969940107 +0000 UTC m=+916.785266594"
Oct 30 12:51:13.536347 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:13.536312    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:15.315420 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:15.315364    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:17.101209 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:17.101174    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:18.704340 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:18.704308    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:19.568933 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:19.568883    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:20.413312 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:20.413279    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:21.133662 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:51:21.133629    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:51:22.099752 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:22.099722    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:23.710132 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:23.710097    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:23.938474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:23.938447    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:51:24.011288 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:24.011213    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"]
Oct 30 12:51:25.386914 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:25.386879    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:27.092844 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:27.092810    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:27.612096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:27.612065    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:51:29.012084 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:29.012027    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df" podUID="9419a58f-4844-4c6c-a208-eb64b2785447" containerName="payload-processing" containerID="cri-o://c9b90cb835ae0608be10d6a34c0e7ce390f8cd527d21bc832ac4d640b4c0e13e" gracePeriod=25
Oct 30 12:51:29.115592 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:29.103403    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:29.235822 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:29.235778    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:30.485601 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.483300    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:30.679959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.525918    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:30.763437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.560816    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:30.763437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.584744    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:30.803479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.777849    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:30.803479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.783197    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:51:30.890106 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.890068    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-routing-overlay-volume\") pod \"9419a58f-4844-4c6c-a208-eb64b2785447\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") "
Oct 30 12:51:30.890618 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.890590    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-plugins-config-volume\") pod \"9419a58f-4844-4c6c-a208-eb64b2785447\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") "
Oct 30 12:51:30.890791 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.890776    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-kube-api-access-czcld\" (UniqueName: \"kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-kube-api-access-czcld\") pod \"9419a58f-4844-4c6c-a208-eb64b2785447\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") "
Oct 30 12:51:30.909527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.909488    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-provider-credentials\") pod \"9419a58f-4844-4c6c-a208-eb64b2785447\" (UID: \"9419a58f-4844-4c6c-a208-eb64b2785447\") "
Oct 30 12:51:30.919428 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.907634    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-plugins-config-volume" pod "9419a58f-4844-4c6c-a208-eb64b2785447" (UID: "9419a58f-4844-4c6c-a208-eb64b2785447"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:51:30.920665 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.920635    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:51:30.964474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.925591    4522 generic.go:356] "Generic (PLEG): container finished" podID="9419a58f-4844-4c6c-a208-eb64b2785447" containerID="c9b90cb835ae0608be10d6a34c0e7ce390f8cd527d21bc832ac4d640b4c0e13e" exitCode=0
Oct 30 12:51:30.964730 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.925636    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df" event={"ID":"9419a58f-4844-4c6c-a208-eb64b2785447","Type":"ContainerDied","Data":"c9b90cb835ae0608be10d6a34c0e7ce390f8cd527d21bc832ac4d640b4c0e13e"}
Oct 30 12:51:30.967613 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.967583    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df" event={"ID":"9419a58f-4844-4c6c-a208-eb64b2785447","Type":"ContainerDied","Data":"9aa04d321a8c64c2b1ae9479b3edc21d59a934d64e9f5b4cb23fa3334df9d448"}
Oct 30 12:51:30.968144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.925716    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"
Oct 30 12:51:30.998491 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.930545    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-routing-overlay-volume" pod "9419a58f-4844-4c6c-a208-eb64b2785447" (UID: "9419a58f-4844-4c6c-a208-eb64b2785447"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:51:30.998714 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.968309    4522 scope.go:122] "RemoveContainer" containerID="c9b90cb835ae0608be10d6a34c0e7ce390f8cd527d21bc832ac4d640b4c0e13e"
Oct 30 12:51:31.011377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:30.925758    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:31.132176 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.132117    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-provider-credentials" pod "9419a58f-4844-4c6c-a208-eb64b2785447" (UID: "9419a58f-4844-4c6c-a208-eb64b2785447"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:51:31.245424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.237144    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-kube-api-access-czcld" pod "9419a58f-4844-4c6c-a208-eb64b2785447" (UID: "9419a58f-4844-4c6c-a208-eb64b2785447"). InnerVolumeSpecName "kube-api-access-czcld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:51:31.254601 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.254565    4522 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9419a58f-4844-4c6c-a208-eb64b2785447-routing-overlay-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:51:31.254797 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.254781    4522 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-provider-credentials\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:51:31.432380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.384176    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czcld\" (UniqueName: \"kubernetes.io/projected/9419a58f-4844-4c6c-a208-eb64b2785447-kube-api-access-czcld\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:51:31.449375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.401729    4522 scope.go:122] "RemoveContainer" containerID="c9b90cb835ae0608be10d6a34c0e7ce390f8cd527d21bc832ac4d640b4c0e13e"
Oct 30 12:51:31.474378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.450774    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:31.508629 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:51:31.458531    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b90cb835ae0608be10d6a34c0e7ce390f8cd527d21bc832ac4d640b4c0e13e\": container with ID starting with c9b90cb835ae0608be10d6a34c0e7ce390f8cd527d21bc832ac4d640b4c0e13e not found: ID does not exist" containerID="c9b90cb835ae0608be10d6a34c0e7ce390f8cd527d21bc832ac4d640b4c0e13e"
Oct 30 12:51:31.509204 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.509167    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b90cb835ae0608be10d6a34c0e7ce390f8cd527d21bc832ac4d640b4c0e13e"} err="failed to get container status \"c9b90cb835ae0608be10d6a34c0e7ce390f8cd527d21bc832ac4d640b4c0e13e\": rpc error: code = NotFound desc = could not find container \"c9b90cb835ae0608be10d6a34c0e7ce390f8cd527d21bc832ac4d640b4c0e13e\": container with ID starting with c9b90cb835ae0608be10d6a34c0e7ce390f8cd527d21bc832ac4d640b4c0e13e not found: ID does not exist"
Oct 30 12:51:31.509333 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.504637    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:31.511195 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.484474    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:31.512502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.506602    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:31.512760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.512741    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:51:31.524124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.523610    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:31.524124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.523678    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:31.761556 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.761448    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:31.762151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.762128    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:51:31.763808 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.763778    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:31.786805 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.782273    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:51:31.819378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.781465    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:31.819378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.781842    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:31.819378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.781875    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:51:31.819378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.789335    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:51:31.819378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.789400    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:51:31.819378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.804432    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:31.819378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.804499    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:31.912388 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.904245    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"]
Oct 30 12:51:31.944384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:31.938545    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-79d5cf7c65-qm5df"]
Oct 30 12:51:32.030385 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.027489    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:32.044375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.036546    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:51:32.490435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.465169    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:32.650514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.650466    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:32.656584 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.569526    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:51:32.684304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.596572    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded"
Oct 30 12:51:32.684559 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.569483    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:32.685156 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.672538    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9419a58f-4844-4c6c-a208-eb64b2785447" path="/var/lib/kubelet/pods/9419a58f-4844-4c6c-a208-eb64b2785447/volumes"
Oct 30 12:51:32.703545 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:51:32.682723    4522 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9419a58f_4844_4c6c_a208_eb64b2785447.slice/crio-9aa04d321a8c64c2b1ae9479b3edc21d59a934d64e9f5b4cb23fa3334df9d448\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9419a58f_4844_4c6c_a208_eb64b2785447.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 12:51:32.783437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.747481    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:51:32.811730 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.785569    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:32.813000 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.812964    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:32.839378 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:51:32.814422    4522 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9419a58f_4844_4c6c_a208_eb64b2785447.slice/crio-9aa04d321a8c64c2b1ae9479b3edc21d59a934d64e9f5b4cb23fa3334df9d448\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9419a58f_4844_4c6c_a208_eb64b2785447.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 12:51:32.841794 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.829700    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:32.843014 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.829731    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:51:32.864122 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.829750    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:51:32.865461 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.829793    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:32.866681 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.866660    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:51:32.949489 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:32.944554    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:33.045054 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:33.044961    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:51:33.113872 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:33.113834    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:51:33.114152 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:33.114121    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:33.688858 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:33.688037    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:33.748608 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:33.747462    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:33.809537 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:33.809450    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:51:33.843520 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:33.843473    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:33.844201 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:33.844141    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:51:33.893786 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:33.893742    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:33.894397 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:33.894021    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:33.933816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:33.933683    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:34.053639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:34.052553    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:51:34.234492 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:34.234443    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:34.698385 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:34.689276    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:51:34.764568 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:34.764515    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:34.823397 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:34.814479    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:51:34.845831 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:34.845785    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:34.847585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:34.847552    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:34.903384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:34.897449    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:34.903384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:34.900474    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:51:35.056380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:35.020729    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Oct 30 12:51:35.057034 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:35.056992    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:35.585967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:35.584940    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:51:35.638550 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:35.638518    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:51:35.771003 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:35.713151    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:35.799376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:35.780894    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:35.894425 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:35.883528    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:35.894425 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:35.883427    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:51:35.898603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:35.898565    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:35.927828 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:35.927469    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:36.023086 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:36.022106    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:51:36.025926 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:36.025901    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:51:36.026891 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:36.026870    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:51:36.057595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:36.057568    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:51:36.069225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:36.069120    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:51:36.104589 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:36.104525    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:36.125629 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:36.125573    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:51:37.802733 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:37.802697    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:39.622161 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:39.622069    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:41.254691 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:41.254658    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:41.396255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:41.396219    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:43.224796 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:43.224764    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:45.018737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:45.018706    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:46.708238 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:46.708201    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:47.668204 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:47.668170    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:48.306763 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:48.306728    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:50.088706 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:50.088671    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:51.135044 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:51:51.135007    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:51:51.805153 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:51.805122    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:53.149246 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.149210    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"]
Oct 30 12:51:53.150135 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.150107    4522 cpu_manager.go:414] "containerMap: removing container" podUID="9419a58f-4844-4c6c-a208-eb64b2785447" containerName="payload-processing"
Oct 30 12:51:53.150135 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.150135    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9419a58f-4844-4c6c-a208-eb64b2785447" containerName="payload-processing"
Oct 30 12:51:53.150309 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.150263    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9419a58f-4844-4c6c-a208-eb64b2785447" containerName="payload-processing"
Oct 30 12:51:53.154656 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.154630    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.175401 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.175378    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"]
Oct 30 12:51:53.180872 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.180840    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c81f42\"/\"distinct-w3-c81f42-kserve-self-signed-certs\""
Oct 30 12:51:53.188571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.188527    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:53.329619 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.329586    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-home\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.330300 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.329690    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npjd2\" (UniqueName: \"kubernetes.io/projected/849bf94f-d57d-4f16-8494-09bbbf8e3db9-kube-api-access-npjd2\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.330300 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.329720    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-dshm\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.330300 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.329778    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tmp-dir\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.330300 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.329799    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tls-certs\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.330300 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.329843    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-model-cache\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.451568 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.451482    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tmp-dir\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.459481 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.459439    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tls-certs\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.461679 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.459755    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-model-cache\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.461938 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.452332    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tmp-dir\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.477478 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.462235    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-home\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.477961 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.477932    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npjd2\" (UniqueName: \"kubernetes.io/projected/849bf94f-d57d-4f16-8494-09bbbf8e3db9-kube-api-access-npjd2\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.478147 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.478129    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-dshm\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.479186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.479159    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tls-certs\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.479421 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.462532    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-home\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.479562 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.462743    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-model-cache\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.500892 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.500857    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-dshm\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.507625 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.507174    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npjd2\" (UniqueName: \"kubernetes.io/projected/849bf94f-d57d-4f16-8494-09bbbf8e3db9-kube-api-access-npjd2\") pod \"distinct-w3-c81f42-kserve-855cb9f656-nt4pb\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:53.672225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.672194    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:53.796840 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:53.796734    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:54.146598 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:51:54.146569    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod849bf94f_d57d_4f16_8494_09bbbf8e3db9.slice/crio-adf8872d0beeff685b6774f06db745f81e7b7ed53a936eab32419fcefa38b53c WatchSource:0}: Error finding container adf8872d0beeff685b6774f06db745f81e7b7ed53a936eab32419fcefa38b53c: Status 404 returned error can't find the container with id adf8872d0beeff685b6774f06db745f81e7b7ed53a936eab32419fcefa38b53c
Oct 30 12:51:54.148229 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:54.148205    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"]
Oct 30 12:51:54.733420 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:54.733384    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" event={"ID":"849bf94f-d57d-4f16-8494-09bbbf8e3db9","Type":"ContainerStarted","Data":"3e69ea35646f68e0bf199206f436fa45d5fb9cffb31de729d3baa7a1fcd9d583"}
Oct 30 12:51:54.733844 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:54.733432    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" event={"ID":"849bf94f-d57d-4f16-8494-09bbbf8e3db9","Type":"ContainerStarted","Data":"adf8872d0beeff685b6774f06db745f81e7b7ed53a936eab32419fcefa38b53c"}
Oct 30 12:51:54.773186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:54.773141    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podStartSLOduration=1.7731289719999999 podStartE2EDuration="1.773128972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:54.770261953 +0000 UTC m=+958.585588442" watchObservedRunningTime="2026-10-30 12:51:54.773128972 +0000 UTC m=+958.588455459"
Oct 30 12:51:54.797625 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:54.797605    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:51:55.425954 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:55.425922    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:56.891476 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:56.891409    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:51:57.121978 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:57.121950    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:59.756374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:59.755527    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:59.760375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:59.758346    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:59.821417 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:59.820439    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:59.855149 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:59.853459    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:51:59.855149 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:59.853822    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:51:59.906563 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:51:59.906509    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:00.065380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:00.064463    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:00.113563 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:00.113520    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:00.710530 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:00.710483    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:00.803376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:00.759484    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:52:00.803376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:00.760218    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:52:00.824903 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:00.822493    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:00.824903 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:00.823190    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:52:00.865380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:00.858752    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:00.865380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:00.859158    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:00.865380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:00.859833    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:52:00.865380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:00.859913    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:52:00.948525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:00.911814    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:52:00.948525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:00.912191    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:52:01.092530 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.092483    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:52:01.163744 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.118467    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:01.175382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.166827    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:52:01.175733 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.175703    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:52:01.247433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.239870    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 30 12:52:01.286508 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.269485    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:52:01.286508 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.269552    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:01.561388 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.556605    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:01.720745 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.715188    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:01.720745 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.716724    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:52:01.762377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.762317    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:01.798845 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.798809    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:52:01.799128 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.799096    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:01.825069 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.823922    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:01.878216 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.878172    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:02.090390 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.882435    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:02.091778 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:01.955851    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:02.165925 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:02.164809    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:02.180378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:02.171426    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:02.400794 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:02.371464    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:52:02.400794 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:02.371528    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:02.554496 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:02.554450    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:02.778435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:02.771594    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:02.866377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:02.840479    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:02.866377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:02.840557    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:52:03.092433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:03.090547    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:52:03.092433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:03.090700    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:52:03.092433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:03.092316    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:03.170377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:03.168676    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:52:03.175384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:03.174566    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:03.775479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:03.775432    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:03.803268 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:03.801913    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:52:03.843479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:03.843423    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:52:03.844677 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:03.844651    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:52:04.125431 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:04.106444    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:04.125431 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:04.106850    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:52:04.125431 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:04.107172    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:52:04.180658 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:04.180619    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:04.227779 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:04.180888    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:04.266310 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:04.226455    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:04.549418 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:04.549388    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:52:04.798503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:04.798454    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:04.828816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:04.828717    4522 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded"
Oct 30 12:52:04.851478 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:04.851435    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:04.871387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:04.854589    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:52:04.982232 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:04.982185    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:05.042376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:05.038827    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:05.111451 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:05.108428    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:52:05.111451 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:05.110648    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:52:05.112002 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:05.111976    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:52:05.156833 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:05.155735    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:52:05.355377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:05.348878    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:52:05.478808 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:05.478726    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:52:05.479595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:05.479576    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:52:05.625392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:05.623697    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:52:05.632165 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:05.632125    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:52:05.648443 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:05.647841    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:52:06.917250 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:06.917216    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:08.693691 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:08.693656    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:10.323116 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:10.323083    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:12.001185 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:12.001150    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:13.736208 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:13.736179    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:13.861373 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:13.858269    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:52:13.940304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:13.940272    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:52:15.432205 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:15.432140    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:16.319888 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:16.319857    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:16.420292 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:16.420261    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:17.302366 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:17.302319    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:19.015431 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:19.015395    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:20.772088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:20.772058    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:21.134675 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:52:21.134595    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:52:22.559207 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:22.559177    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:24.219662 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:24.219627    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:24.743586 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:24.743543    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:24.846725 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:24.846699    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:26.110677 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:26.110643    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:27.876486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:27.876447    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:29.261680 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:29.261650    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:52:29.624952 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:29.624918    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:31.990832 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:31.990795    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:32.877376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:32.870087    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:33.098383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:33.093244    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:52:33.829530 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:33.829477    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:33.859387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:33.858436    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:33.875713 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:33.872427    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:52:33.875713 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:33.873890    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:52:34.267474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.201056    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:52:34.267474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.238368    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:52:34.267474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.238733    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:52:34.267474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.239329    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:52:34.267474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.254525    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:34.267474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.255406    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 30 12:52:34.267474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.266649    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:34.794379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.793531    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:34.812483 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.812434    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:34.851492 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.851445    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:52:34.853144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.853118    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:52:34.879722 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.879679    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:34.880220 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.880156    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:52:34.888374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:34.881250    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:52:35.211573 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.211467    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:35.214618 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.214574    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:35.215164 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.215135    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:52:35.272603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.272555    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:52:35.299376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.274584    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:52:35.302337 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.302299    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:52:35.335759 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.335729    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:52:35.364422 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.338837    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:52:35.391402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.358803    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:52:35.391402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.388243    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:52:35.440742 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.440706    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:52:35.648382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.640891    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:35.802374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.777577    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:35.805339 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.794254    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:35.806252 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.806229    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:52:35.825376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.821282    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:35.929126 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.855505    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:35.933375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.898645    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:52:35.933867 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:35.898691    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:52:36.186580 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:36.186485    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:52:36.195384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:36.186868    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:36.305413 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:36.275615    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:36.347620 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:36.347550    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:36.399907 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:36.399863    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:52:36.421504 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:36.421443    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:52:36.427383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:36.421836    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:52:36.898375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:36.897401    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:52:36.898375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:36.897781    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:52:36.898375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:36.897831    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:36.954453 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:36.947346    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:36.954453 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:36.947864    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:52:36.981104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:36.981061    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:37.280469 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:37.280425    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:37.354514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:37.354470    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:37.406547 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:37.406491    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:52:37.575436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:37.571802    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:37.575436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:37.572823    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:37.575436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:37.573003    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:37.902238 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:37.902126    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:37.950528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:37.950482    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:52:38.161428 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.157470    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:38.230495 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.229809    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:38.286073 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.285036    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:38.387240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.387167    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:38.412609 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.412512    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:38.578380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.577458    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:52:38.578380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.578076    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:38.620578 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.620547    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"]
Oct 30 12:52:38.625464 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.625436    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"]
Oct 30 12:52:38.676426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.630904    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:38.686406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.664303    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:52:38.686406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.678581    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"]
Oct 30 12:52:38.706732 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.706231    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c81f42\"/\"distinct-2-w3-c81f42-kserve-self-signed-certs\""
Oct 30 12:52:38.780873 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.780835    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:38.934495 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.918387    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-tmp-dir\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:38.934495 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.918484    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-home\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:38.934495 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.918517    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhbpr\" (UniqueName: \"kubernetes.io/projected/dc1616ab-319c-4608-893b-fdcc65631488-kube-api-access-qhbpr\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:38.934495 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.918638    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-model-cache\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:38.934495 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.918703    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-dshm\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:38.934495 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:38.918866    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1616ab-319c-4608-893b-fdcc65631488-tls-certs\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.014033 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.013989    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:52:39.036543 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.036500    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:52:39.046419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.039523    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-model-cache\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.046419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.039600    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-dshm\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.046419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.039768    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1616ab-319c-4608-893b-fdcc65631488-tls-certs\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.046419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.039873    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-tmp-dir\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.046419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.039921    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-home\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.046419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.039950    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhbpr\" (UniqueName: \"kubernetes.io/projected/dc1616ab-319c-4608-893b-fdcc65631488-kube-api-access-qhbpr\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.046419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.040708    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-model-cache\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.082376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.055620    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-tmp-dir\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.082376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.069848    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1616ab-319c-4608-893b-fdcc65631488-tls-certs\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.093944 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.083584    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-home\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.209308 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.209211    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:52:39.286094 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.268368    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:39.289031 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.288956    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-dshm\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.336606 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.286418    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:52:39.344742 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.344585    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:52:39.373560 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.361596    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:52:39.373560 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.372890    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhbpr\" (UniqueName: \"kubernetes.io/projected/dc1616ab-319c-4608-893b-fdcc65631488-kube-api-access-qhbpr\") pod \"distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") " pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.390500 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.389760    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:52:39.390500 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.389816    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:52:39.395708 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.395689    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:52:39.405799 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.404179    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:52:39.405799 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.405642    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:52:39.449704 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.449681    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:39.526576 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.526546    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:39.875380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:39.869438    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"]
Oct 30 12:52:40.344922 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:40.344892    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" event={"ID":"dc1616ab-319c-4608-893b-fdcc65631488","Type":"ContainerStarted","Data":"f4c954dc8c41bae46b3c16842bc1cf8917474a372a0c1e2eba872295b96bc7bc"}
Oct 30 12:52:40.345078 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:40.344928    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" event={"ID":"dc1616ab-319c-4608-893b-fdcc65631488","Type":"ContainerStarted","Data":"123a0c36adf0bb5cfeb4d7cc67d60a3579e2f9cdff55bf8423c96e28e1a6e3ce"}
Oct 30 12:52:40.385437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:40.385395    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podStartSLOduration=2.38538162 podStartE2EDuration="2.38538162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:40.384065835 +0000 UTC m=+1004.199392326" watchObservedRunningTime="2026-10-30 12:52:40.38538162 +0000 UTC m=+1004.200708108"
Oct 30 12:52:40.450653 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:40.450630    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:41.277475 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:41.277434    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:43.007719 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:43.007688    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:43.621787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:43.621749    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk" podUID="1f90c8eb-8b1f-4f4b-a741-e1fae55be35b" containerName="payload-processing" containerID="cri-o://c52d5537ca922f0f0a40559aa9499d552486601d7d5c5cbf6c12c5c0b6d8538b" gracePeriod=25
Oct 30 12:52:43.918981 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:43.918939    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:52:44.016723 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.016694    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-plugins-config-volume\") pod \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") "
Oct 30 12:52:44.017159 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.016775    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-routing-overlay-volume\") pod \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") "
Oct 30 12:52:44.017159 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.016929    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-kube-api-access-zp84w\" (UniqueName: \"kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-kube-api-access-zp84w\") pod \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") "
Oct 30 12:52:44.017159 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.017096    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-plugins-config-volume" pod "1f90c8eb-8b1f-4f4b-a741-e1fae55be35b" (UID: "1f90c8eb-8b1f-4f4b-a741-e1fae55be35b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:44.017325 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.017165    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-routing-overlay-volume" pod "1f90c8eb-8b1f-4f4b-a741-e1fae55be35b" (UID: "1f90c8eb-8b1f-4f4b-a741-e1fae55be35b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:44.017610 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.017588    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:52:44.017693 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.017616    4522 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-routing-overlay-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:52:44.018901 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.018880    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-kube-api-access-zp84w" pod "1f90c8eb-8b1f-4f4b-a741-e1fae55be35b" (UID: "1f90c8eb-8b1f-4f4b-a741-e1fae55be35b"). InnerVolumeSpecName "kube-api-access-zp84w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:44.118087 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.118057    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-provider-credentials\") pod \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\" (UID: \"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b\") "
Oct 30 12:52:44.118571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.118555    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp84w\" (UniqueName: \"kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-kube-api-access-zp84w\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:52:44.120057 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.120036    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-provider-credentials" pod "1f90c8eb-8b1f-4f4b-a741-e1fae55be35b" (UID: "1f90c8eb-8b1f-4f4b-a741-e1fae55be35b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:44.219596 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.219521    4522 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b-provider-credentials\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:52:44.420138 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.420103    4522 generic.go:356] "Generic (PLEG): container finished" podID="1f90c8eb-8b1f-4f4b-a741-e1fae55be35b" containerID="c52d5537ca922f0f0a40559aa9499d552486601d7d5c5cbf6c12c5c0b6d8538b" exitCode=0
Oct 30 12:52:44.420271 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.420148    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"
Oct 30 12:52:44.420271 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.420194    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk" event={"ID":"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b","Type":"ContainerDied","Data":"c52d5537ca922f0f0a40559aa9499d552486601d7d5c5cbf6c12c5c0b6d8538b"}
Oct 30 12:52:44.420271 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.420255    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk" event={"ID":"1f90c8eb-8b1f-4f4b-a741-e1fae55be35b","Type":"ContainerDied","Data":"6550e3e6e9a373412a8eda029b96b6bd5d077dc600262f865f54a18615bf2fae"}
Oct 30 12:52:44.420413 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.420287    4522 scope.go:122] "RemoveContainer" containerID="c52d5537ca922f0f0a40559aa9499d552486601d7d5c5cbf6c12c5c0b6d8538b"
Oct 30 12:52:44.430602 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.430582    4522 scope.go:122] "RemoveContainer" containerID="c52d5537ca922f0f0a40559aa9499d552486601d7d5c5cbf6c12c5c0b6d8538b"
Oct 30 12:52:44.430886 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:52:44.430868    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c52d5537ca922f0f0a40559aa9499d552486601d7d5c5cbf6c12c5c0b6d8538b\": container with ID starting with c52d5537ca922f0f0a40559aa9499d552486601d7d5c5cbf6c12c5c0b6d8538b not found: ID does not exist" containerID="c52d5537ca922f0f0a40559aa9499d552486601d7d5c5cbf6c12c5c0b6d8538b"
Oct 30 12:52:44.430969 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.430894    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c52d5537ca922f0f0a40559aa9499d552486601d7d5c5cbf6c12c5c0b6d8538b"} err="failed to get container status \"c52d5537ca922f0f0a40559aa9499d552486601d7d5c5cbf6c12c5c0b6d8538b\": rpc error: code = NotFound desc = could not find container \"c52d5537ca922f0f0a40559aa9499d552486601d7d5c5cbf6c12c5c0b6d8538b\": container with ID starting with c52d5537ca922f0f0a40559aa9499d552486601d7d5c5cbf6c12c5c0b6d8538b not found: ID does not exist"
Oct 30 12:52:44.470138 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.470082    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"]
Oct 30 12:52:44.481546 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.481527    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d97d47b88-5l9wk"]
Oct 30 12:52:44.703647 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:44.703616    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:46.336145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:46.336115    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f90c8eb-8b1f-4f4b-a741-e1fae55be35b" path="/var/lib/kubelet/pods/1f90c8eb-8b1f-4f4b-a741-e1fae55be35b/volumes"
Oct 30 12:52:46.467258 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:46.467228    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:47.197448 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:47.197414    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:47.297118 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:47.297082    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:47.828246 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:47.828214    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:48.416293 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:48.416259    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:49.440465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:49.440399    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:49.453775 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:49.453751    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:49.467327 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:49.467303    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:52:50.109247 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:50.109214    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:51.133409 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:52:51.133380    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:52:51.729307 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:51.729273    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:53.519965 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:53.519933    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:55.207283 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:55.207248    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:56.997208 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:56.997126    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:58.264273 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:58.264239    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:58.922205 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:52:58.922171    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:00.706223 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:00.706192    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:02.397321 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:02.397290    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:04.070489 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:04.070457    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:05.820080 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:05.820033    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:07.425009 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:07.424980    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:09.022677 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:09.022585    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:09.188506 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:09.187632    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:10.701634 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:10.701601    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:11.660400 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:11.660344    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:53:12.465848 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:12.465783    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:14.427468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:14.427419    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:53:14.428380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:14.428040    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:53:14.813411 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:14.798433    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:14.889379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:14.884656    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:14.905390 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:14.904439    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:14.956403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:14.951490    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:14.976396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:14.966450    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:53:14.979489 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:14.979441    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:15.288427 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.279437    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:15.359470 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.358487    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:15.418486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.418441    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:15.448376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.430420    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:15.450712 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.449131    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:53:15.450712 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.449681    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:53:15.450712 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.449739    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:53:15.503669 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.503622    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:53:15.774824 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.762481    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded"
Oct 30 12:53:15.837375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.827046    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:15.889742 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.889700    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:53:15.891088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.891063    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:53:15.913793 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.913754    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:15.914322 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.914299    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:53:16.017379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.991450    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:16.017379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:15.992035    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:53:16.083450 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:16.069435    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:53:16.083450 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:16.069989    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:53:16.299379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:16.281485    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:53:16.322278 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:16.322250    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:53:16.412436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:16.410747    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:16.549376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:16.422145    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:16.748871 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:16.702597    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:53:16.963278 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:16.963236    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:53:17.061492 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:16.989482    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:53:17.113517 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:17.113470    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:17.192378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:17.188927    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:53:17.224804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:17.190537    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:53:17.225083 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:17.223698    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:53:17.350200 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:17.317525    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:17.626426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:17.591118    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:53:18.017131 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:17.598414    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 12:53:18.059047 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:53:17.789734    4522 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.409s"
Oct 30 12:53:18.059502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:18.059477    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:53:18.059747 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:18.059734    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:53:18.059933 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:18.059905    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:53:18.061136 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:18.027519    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:53:18.063587 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:17.951821    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:18.063587 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:17.991474    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:18.063587 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:18.027467    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:18.125707 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:18.125663    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:18.238398 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:18.229433    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:18.238398 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:18.229798    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:53:18.238398 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:18.230060    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:53:18.326476 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:18.326422    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:19.030169 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:19.030112    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:19.072329 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:19.072276    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:19.072716 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:19.072695    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:53:19.073688 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:19.072584    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:19.083450 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:19.072227    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:53:19.145377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:19.136474    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:19.238532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:19.238482    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:19.240307 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:19.238779    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:53:19.241497 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:19.238805    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:53:19.334554 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:19.334450    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:19.519131 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:19.519071    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:20.038812 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:20.038755    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:53:20.075434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:20.074436    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:20.075716 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:20.075686    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:53:20.076032 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:20.075892    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:20.099125 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:20.098594    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:53:20.138424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:20.137448    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:20.244468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:20.244425    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:53:20.245182 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:20.245135    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:20.245631 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:20.245605    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:20.354564 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:20.354452    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:20.464678 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:20.464642    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded"
Oct 30 12:53:21.044377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:21.043104    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:21.078376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:21.077434    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:21.082380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:21.078565    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:53:21.082380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:21.078856    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:21.114389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:21.111430    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:21.149707 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:21.149657    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:53:21.172517 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:21.172477    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 30 12:53:21.246650 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:21.246604    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:21.247241 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:21.247205    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:53:21.247684 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:21.247659    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:53:21.366584 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:21.366482    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:21.554377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:21.529516    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded"
Oct 30 12:53:21.672896 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:53:21.642660    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:53:22.061577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.061536    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:53:22.087083 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.084462    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:22.087083 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.084858    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:22.090969 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.090173    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:22.114674 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.114142    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:22.178895 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.176514    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:53:22.192423 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.192395    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:53:22.250467 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.240234    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:22.250981 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.250900    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:22.252665 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.252633    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:53:22.268375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.263316    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:53:22.278385 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.272331    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:53:22.282330 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.282297    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:22.283164 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.283126    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:53:22.373873 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.373796    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:53:22.374114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.374098    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:53:22.374240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.374231    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:53:22.374382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:22.374371    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:53:23.806615 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:23.806585    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:25.513287 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:25.513248    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:27.116590 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:27.116554    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:28.798939 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:28.798894    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:30.400027 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:30.399993    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:32.007293 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:32.007258    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:33.707981 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:33.707946    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:35.437768 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:35.437736    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:36.344469 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.344439    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"]
Oct 30 12:53:36.345069 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.345017    4522 cpu_manager.go:414] "containerMap: removing container" podUID="1f90c8eb-8b1f-4f4b-a741-e1fae55be35b" containerName="payload-processing"
Oct 30 12:53:36.345069 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.345037    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f90c8eb-8b1f-4f4b-a741-e1fae55be35b" containerName="payload-processing"
Oct 30 12:53:36.345277 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.345209    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f90c8eb-8b1f-4f4b-a741-e1fae55be35b" containerName="payload-processing"
Oct 30 12:53:36.352378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.352336    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.353394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.353347    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"]
Oct 30 12:53:36.366496 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.366468    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:36.378185 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.378168    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-53dcb1\"/\"distinct-2-w4-53dcb1-kserve-self-signed-certs\""
Oct 30 12:53:36.447457 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.447432    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c562c6-1c99-4494-9436-137188543b90-tls-certs\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.447837 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.447473    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq2pv\" (UniqueName: \"kubernetes.io/projected/29c562c6-1c99-4494-9436-137188543b90-kube-api-access-gq2pv\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.447837 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.447612    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-home\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.447837 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.447687    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-model-cache\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.447837 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.447716    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-tmp-dir\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.447837 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.447791    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-dshm\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.558375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.556328    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-tmp-dir\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.558375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.556431    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-dshm\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.558375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.556661    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c562c6-1c99-4494-9436-137188543b90-tls-certs\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.558375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.556696    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq2pv\" (UniqueName: \"kubernetes.io/projected/29c562c6-1c99-4494-9436-137188543b90-kube-api-access-gq2pv\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.558375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.556768    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-home\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.558375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.556811    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-model-cache\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.558375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.557255    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-model-cache\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.558375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.557525    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-tmp-dir\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.580372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.572912    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-home\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.580372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.574148    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c562c6-1c99-4494-9436-137188543b90-tls-certs\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.583559 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.583525    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-dshm\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.596419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.591968    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq2pv\" (UniqueName: \"kubernetes.io/projected/29c562c6-1c99-4494-9436-137188543b90-kube-api-access-gq2pv\") pod \"distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") " pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:36.672430 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:36.671719    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:37.049501 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:53:37.049468    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29c562c6_1c99_4494_9436_137188543b90.slice/crio-4268ef896f413d36f1efceb3c3e674e89e8c31a62f3573d77e66522ed4b41327 WatchSource:0}: Error finding container 4268ef896f413d36f1efceb3c3e674e89e8c31a62f3573d77e66522ed4b41327: Status 404 returned error can't find the container with id 4268ef896f413d36f1efceb3c3e674e89e8c31a62f3573d77e66522ed4b41327
Oct 30 12:53:37.049867 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:37.049847    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"]
Oct 30 12:53:37.263719 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:37.263688    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:37.453713 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:37.453611    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" event={"ID":"29c562c6-1c99-4494-9436-137188543b90","Type":"ContainerStarted","Data":"fd6262173ff2c2ddab9746adf517d6dbc7a7ce879d203c8e760827673abd7409"}
Oct 30 12:53:37.453713 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:37.453691    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" event={"ID":"29c562c6-1c99-4494-9436-137188543b90","Type":"ContainerStarted","Data":"4268ef896f413d36f1efceb3c3e674e89e8c31a62f3573d77e66522ed4b41327"}
Oct 30 12:53:37.493101 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:37.493052    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podStartSLOduration=1.493038829 podStartE2EDuration="1.493038829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:37.492465349 +0000 UTC m=+1061.307791840" watchObservedRunningTime="2026-10-30 12:53:37.493038829 +0000 UTC m=+1061.308365349"
Oct 30 12:53:37.673032 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:37.672996    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:38.918458 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:38.918424    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:40.711977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:40.711944    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:42.295976 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:42.295944    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:44.010210 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:44.010178    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:45.502975 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:45.502922    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:45.628549 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:45.628517    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:46.017968 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:46.017927    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:46.130036 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:46.130007    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:46.672342 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:46.672309    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:46.684563 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:46.684539    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:47.500657 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:47.500611    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:47.527475 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:47.527443    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:53:47.946070 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:47.946035    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:49.127589 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:49.127557    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:49.286456 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:49.286378    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:49.390978 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:49.390930    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:49.502451 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:49.502416    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:50.895183 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:50.895128    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:50.995746 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:50.995713    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:51.071368 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.071325    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"]
Oct 30 12:53:51.075379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.075344    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"
Oct 30 12:53:51.093078 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.093054    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"]
Oct 30 12:53:51.100241 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.100218    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7a04e002-dockercfg-zcb8n\""
Oct 30 12:53:51.100431 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.100246    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7a04e002\""
Oct 30 12:53:51.224449 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:53:51.224342    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:53:51.366029 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.365158    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"]
Oct 30 12:53:51.376547 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.376506    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:51.439935 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.439872    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"
Oct 30 12:53:51.614114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.613414    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tngtp\" (UniqueName: \"kubernetes.io/projected/08206131-6355-4479-bfaf-a9920a03bc91-kube-api-access-tngtp\") pod \"payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8\" (UID: \"08206131-6355-4479-bfaf-a9920a03bc91\") " pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"
Oct 30 12:53:51.614114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.613546    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08206131-6355-4479-bfaf-a9920a03bc91-plugins-config-volume\") pod \"payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8\" (UID: \"08206131-6355-4479-bfaf-a9920a03bc91\") " pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"
Oct 30 12:53:51.614114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.613588    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq\" (UID: \"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"
Oct 30 12:53:51.614114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.613667    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t24j\" (UniqueName: \"kubernetes.io/projected/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-kube-api-access-4t24j\") pod \"payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq\" (UID: \"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"
Oct 30 12:53:51.617458 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.617018    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"]
Oct 30 12:53:51.715009 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.714983    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t24j\" (UniqueName: \"kubernetes.io/projected/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-kube-api-access-4t24j\") pod \"payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq\" (UID: \"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"
Oct 30 12:53:51.715248 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.715217    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tngtp\" (UniqueName: \"kubernetes.io/projected/08206131-6355-4479-bfaf-a9920a03bc91-kube-api-access-tngtp\") pod \"payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8\" (UID: \"08206131-6355-4479-bfaf-a9920a03bc91\") " pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"
Oct 30 12:53:51.715401 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.715325    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08206131-6355-4479-bfaf-a9920a03bc91-plugins-config-volume\") pod \"payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8\" (UID: \"08206131-6355-4479-bfaf-a9920a03bc91\") " pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"
Oct 30 12:53:51.715401 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.715381    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq\" (UID: \"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"
Oct 30 12:53:51.716076 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.716050    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08206131-6355-4479-bfaf-a9920a03bc91-plugins-config-volume\") pod \"payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8\" (UID: \"08206131-6355-4479-bfaf-a9920a03bc91\") " pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"
Oct 30 12:53:51.716181 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.716064    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq\" (UID: \"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"
Oct 30 12:53:51.734654 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.734631    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t24j\" (UniqueName: \"kubernetes.io/projected/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-kube-api-access-4t24j\") pod \"payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq\" (UID: \"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"
Oct 30 12:53:51.734921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.734893    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tngtp\" (UniqueName: \"kubernetes.io/projected/08206131-6355-4479-bfaf-a9920a03bc91-kube-api-access-tngtp\") pod \"payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8\" (UID: \"08206131-6355-4479-bfaf-a9920a03bc91\") " pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"
Oct 30 12:53:51.928987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.928904    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"
Oct 30 12:53:51.990293 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:51.989777    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"
Oct 30 12:53:52.228535 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:53:52.228493    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08206131_6355_4479_bfaf_a9920a03bc91.slice/crio-2edbe9bf6bc479d605701e7d2d141d2a0abf79ec541d2cbc2a3e8be288325012 WatchSource:0}: Error finding container 2edbe9bf6bc479d605701e7d2d141d2a0abf79ec541d2cbc2a3e8be288325012: Status 404 returned error can't find the container with id 2edbe9bf6bc479d605701e7d2d141d2a0abf79ec541d2cbc2a3e8be288325012
Oct 30 12:53:52.228674 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:52.228653    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"]
Oct 30 12:53:52.298024 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:53:52.297995    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fd5c1b_b147_44be_b82a_a69fc5f2ee11.slice/crio-25c35e703082eaca6b6a654e867a7b0864773d4664a28662928099ec7bf49378 WatchSource:0}: Error finding container 25c35e703082eaca6b6a654e867a7b0864773d4664a28662928099ec7bf49378: Status 404 returned error can't find the container with id 25c35e703082eaca6b6a654e867a7b0864773d4664a28662928099ec7bf49378
Oct 30 12:53:52.299007 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:52.298980    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"]
Oct 30 12:53:52.631708 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:52.631668    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8" event={"ID":"08206131-6355-4479-bfaf-a9920a03bc91","Type":"ContainerStarted","Data":"31e39cd9cd0badefc6d7c0c8255ac25662b933a7afc43a5f50289ae81590c61b"}
Oct 30 12:53:52.631875 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:52.631848    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"
Oct 30 12:53:52.631924 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:52.631891    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8" event={"ID":"08206131-6355-4479-bfaf-a9920a03bc91","Type":"ContainerStarted","Data":"2edbe9bf6bc479d605701e7d2d141d2a0abf79ec541d2cbc2a3e8be288325012"}
Oct 30 12:53:52.633312 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:52.633285    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq" event={"ID":"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11","Type":"ContainerStarted","Data":"d100541ef2f5aaae7fdff75adb3999203890dc44e019981d9c619b9280767a1f"}
Oct 30 12:53:52.633434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:52.633335    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq" event={"ID":"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11","Type":"ContainerStarted","Data":"25c35e703082eaca6b6a654e867a7b0864773d4664a28662928099ec7bf49378"}
Oct 30 12:53:52.633434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:52.633394    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"
Oct 30 12:53:52.670996 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:52.670938    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8" podStartSLOduration=1.670925043 podStartE2EDuration="1.670925043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:52.66822428 +0000 UTC m=+1076.483550770" watchObservedRunningTime="2026-10-30 12:53:52.670925043 +0000 UTC m=+1076.486251509"
Oct 30 12:53:52.708970 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:52.708932    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq" podStartSLOduration=1.7089204420000002 podStartE2EDuration="1.708920442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:52.705950897 +0000 UTC m=+1076.521277387" watchObservedRunningTime="2026-10-30 12:53:52.708920442 +0000 UTC m=+1076.524246966"
Oct 30 12:53:53.007638 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:53.007544    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:54.698911 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:54.698868    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:56.301715 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:56.301684    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:56.780944 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:56.780909    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:53:57.512138 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:57.512107    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:58.035554 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:58.025221    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:58.185376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:58.174132    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:58.264411 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:53:58.264379    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:00.056512 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:00.056457    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:00.084753 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:00.084700    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:54:00.106498 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:00.106454    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:54:00.106869 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:00.106835    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:00.348429 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:00.347575    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:00.457377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:00.456447    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:00.525470 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:00.525423    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:54:01.047378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.041980    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:54:01.063377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.059438    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:01.063377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.060193    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:54:01.098387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.091604    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:01.098387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.092183    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:54:01.130804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.117450    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:01.130804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.117965    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:54:01.130804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.118965    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:54:01.130804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.119432    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:54:01.134111 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.131251    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:01.240386 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.199610    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:01.240386 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.231450    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:01.240386 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.231970    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:01.314263 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.310918    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:01.348420 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.348302    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:01.358376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.348934    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:54:01.516201 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.516146    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:01.533531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.533491    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:01.534327 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:01.534307    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:54:02.054175 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.054043    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:02.100164 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.099980    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:02.101554 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.100027    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:02.102107 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.100984    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:54:02.137217 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.135974    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:02.157824 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.137519    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:54:02.206393 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.160495    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:54:02.206393 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.160870    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:54:02.250381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.232894    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:54:02.250381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.233158    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:54:02.250381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.233579    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:54:02.250381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.233636    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:54:02.317210 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.316954    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:02.318766 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.318743    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:54:02.363468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.363424    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:54:02.535381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:02.534431    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:03.053383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.049221    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:54:03.053383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.049320    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:54:03.128379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.115921    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:03.128379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.116326    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:54:03.128379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.116637    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:03.157022 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.156978    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:03.163687 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.163649    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:54:03.169319 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.169292    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:54:03.242254 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.242210    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:54:03.243015 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.242981    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:54:03.411458 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.393833    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:03.607879 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.607818    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:03.843784 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.843742    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:03.904659 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:03.843346    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"
Oct 30 12:54:04.169258 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.169060    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:04.296096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.195142    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:04.296096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.200434    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:04.296096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.200692    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:54:04.296096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.200771    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:54:04.296096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.201197    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"
Oct 30 12:54:04.296096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.201423    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:04.296096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.288505    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:04.296096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.288550    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 30 12:54:04.296096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.288567    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:54:04.500127 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.500021    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:04.618376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.612475    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:04.618376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.612862    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:54:04.812378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.811430    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:04.864455 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.863590    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:04.864455 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:04.864190    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:54:05.034716 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.034664    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:05.355503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.299384    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:05.355503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.299763    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:54:05.355503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.300037    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:54:05.355503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.300277    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:54:05.355503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.315738    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:05.355503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.316022    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:54:05.355503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.316268    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:54:05.355503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.317442    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:54:05.542377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.542179    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:05.620717 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.619587    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:05.728502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.728449    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:05.865480 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.865434    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:05.952429 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:05.950090    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:06.013898 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.013444    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:54:06.013898 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.013505    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:06.163256 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.163144    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:06.315841 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.298869    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:06.364418 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.302428    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:54:06.384715 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.384669    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:54:06.497642 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.302485    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:54:06.498324 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.302504    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:54:06.502683 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.302465    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:54:06.502832 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.395842    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:54:06.502973 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.395879    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:54:06.510452 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.438737    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:54:06.564381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.547228    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:06.635426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.624706    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:06.707380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.676725    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:06.919425 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:06.918686    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:07.079379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.078066    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:07.208433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.208085    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:07.314967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.314912    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:54:07.393732 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.387247    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:07.423376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.417138    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:07.547441 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.528747    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:07.547441 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.529705    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:07.547441 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.529960    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:54:07.547441 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.530190    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:54:07.571386 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.557456    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:07.571386 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.557826    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:54:07.571386 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.558080    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:54:07.627816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.626696    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:54:07.685171 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.684884    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded"
Oct 30 12:54:07.759406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.731295    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:07.976462 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:07.960274    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:54:08.316414 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:08.315948    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:08.324406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:08.316992    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:54:08.388302 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:08.388265    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:08.416398 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:08.414478    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:54:08.570297 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:08.569564    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:08.592507 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:08.592430    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:54:08.593249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:08.593208    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:54:08.741138 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:08.608216    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:08.928463 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:08.609544    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:09.035067 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:08.609575    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:54:09.035067 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:08.641794    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:54:09.035067 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.002133    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:54:09.035067 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.012720    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:09.070914 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.070879    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:54:09.084889 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.084861    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:54:09.085731 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.085707    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:54:09.088797 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.088676    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:54:09.089267 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.089241    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:54:09.089764 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.089745    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:54:09.091808 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.091775    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:54:09.197492 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.197327    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:09.304949 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.304911    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:09.374672 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.374641    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"]
Oct 30 12:54:09.380167 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.380127    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"
Oct 30 12:54:09.395770 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.395625    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0a531c-dockercfg-prkns\""
Oct 30 12:54:09.395770 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.395729    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0a531c\""
Oct 30 12:54:09.399050 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.399022    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"]
Oct 30 12:54:09.472058 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.471985    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"]
Oct 30 12:54:09.476211 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.476192    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"
Oct 30 12:54:09.491372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.491336    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52ml4\" (UniqueName: \"kubernetes.io/projected/64b94153-ad76-4ce3-8afd-a05398816dd4-kube-api-access-52ml4\") pod \"payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9\" (UID: \"64b94153-ad76-4ce3-8afd-a05398816dd4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"
Oct 30 12:54:09.491496 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.491401    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b94153-ad76-4ce3-8afd-a05398816dd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9\" (UID: \"64b94153-ad76-4ce3-8afd-a05398816dd4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"
Oct 30 12:54:09.496397 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.496375    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"]
Oct 30 12:54:09.594377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.593293    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52ml4\" (UniqueName: \"kubernetes.io/projected/64b94153-ad76-4ce3-8afd-a05398816dd4-kube-api-access-52ml4\") pod \"payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9\" (UID: \"64b94153-ad76-4ce3-8afd-a05398816dd4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"
Oct 30 12:54:09.594377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.593410    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b94153-ad76-4ce3-8afd-a05398816dd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9\" (UID: \"64b94153-ad76-4ce3-8afd-a05398816dd4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"
Oct 30 12:54:09.594377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.593569    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjljs\" (UniqueName: \"kubernetes.io/projected/d2a8150a-f0de-4f31-94de-d44575f13f05-kube-api-access-gjljs\") pod \"payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr\" (UID: \"d2a8150a-f0de-4f31-94de-d44575f13f05\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"
Oct 30 12:54:09.594377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.593618    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a8150a-f0de-4f31-94de-d44575f13f05-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr\" (UID: \"d2a8150a-f0de-4f31-94de-d44575f13f05\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"
Oct 30 12:54:09.594816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.594790    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b94153-ad76-4ce3-8afd-a05398816dd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9\" (UID: \"64b94153-ad76-4ce3-8afd-a05398816dd4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"
Oct 30 12:54:09.636372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.635130    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:09.657095 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.653432    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52ml4\" (UniqueName: \"kubernetes.io/projected/64b94153-ad76-4ce3-8afd-a05398816dd4-kube-api-access-52ml4\") pod \"payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9\" (UID: \"64b94153-ad76-4ce3-8afd-a05398816dd4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"
Oct 30 12:54:09.697665 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.697620    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjljs\" (UniqueName: \"kubernetes.io/projected/d2a8150a-f0de-4f31-94de-d44575f13f05-kube-api-access-gjljs\") pod \"payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr\" (UID: \"d2a8150a-f0de-4f31-94de-d44575f13f05\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"
Oct 30 12:54:09.697977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.697949    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a8150a-f0de-4f31-94de-d44575f13f05-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr\" (UID: \"d2a8150a-f0de-4f31-94de-d44575f13f05\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"
Oct 30 12:54:09.699513 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.699491    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a8150a-f0de-4f31-94de-d44575f13f05-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr\" (UID: \"d2a8150a-f0de-4f31-94de-d44575f13f05\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"
Oct 30 12:54:09.712920 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.712884    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"
Oct 30 12:54:09.721919 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.721894    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjljs\" (UniqueName: \"kubernetes.io/projected/d2a8150a-f0de-4f31-94de-d44575f13f05-kube-api-access-gjljs\") pod \"payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr\" (UID: \"d2a8150a-f0de-4f31-94de-d44575f13f05\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"
Oct 30 12:54:09.817260 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:09.809948    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"
Oct 30 12:54:10.257240 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:54:10.257210    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b94153_ad76_4ce3_8afd_a05398816dd4.slice/crio-5751afc617818f4400b01ec4916f26060365bcc0f84f63b10632addfee2b3b61 WatchSource:0}: Error finding container 5751afc617818f4400b01ec4916f26060365bcc0f84f63b10632addfee2b3b61: Status 404 returned error can't find the container with id 5751afc617818f4400b01ec4916f26060365bcc0f84f63b10632addfee2b3b61
Oct 30 12:54:10.259225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:10.259205    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"]
Oct 30 12:54:10.270849 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:10.270822    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"]
Oct 30 12:54:10.271809 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:54:10.271784    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a8150a_f0de_4f31_94de_d44575f13f05.slice/crio-d663315833fce049eddc16b56c5bda3f0dff849c7972f53e252aadfd9ffd6a3c WatchSource:0}: Error finding container d663315833fce049eddc16b56c5bda3f0dff849c7972f53e252aadfd9ffd6a3c: Status 404 returned error can't find the container with id d663315833fce049eddc16b56c5bda3f0dff849c7972f53e252aadfd9ffd6a3c
Oct 30 12:54:11.223249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:11.223214    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9" event={"ID":"64b94153-ad76-4ce3-8afd-a05398816dd4","Type":"ContainerStarted","Data":"05e4fc573f6062e033ebbb760f21f076ae148f53bb82712bfa7ccdd58ad25814"}
Oct 30 12:54:11.223703 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:11.223273    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9" event={"ID":"64b94153-ad76-4ce3-8afd-a05398816dd4","Type":"ContainerStarted","Data":"5751afc617818f4400b01ec4916f26060365bcc0f84f63b10632addfee2b3b61"}
Oct 30 12:54:11.223703 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:11.223408    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"
Oct 30 12:54:11.224754 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:11.224732    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr" event={"ID":"d2a8150a-f0de-4f31-94de-d44575f13f05","Type":"ContainerStarted","Data":"ab088d07ea34bd23e903017c2024a7a4915457f8cb3590b4a7faf6ea2fecf7f9"}
Oct 30 12:54:11.224859 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:11.224768    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr" event={"ID":"d2a8150a-f0de-4f31-94de-d44575f13f05","Type":"ContainerStarted","Data":"d663315833fce049eddc16b56c5bda3f0dff849c7972f53e252aadfd9ffd6a3c"}
Oct 30 12:54:11.224859 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:11.224792    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"
Oct 30 12:54:11.265777 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:11.265724    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9" podStartSLOduration=2.265710581 podStartE2EDuration="2.265710581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:11.262827618 +0000 UTC m=+1095.078154105" watchObservedRunningTime="2026-10-30 12:54:11.265710581 +0000 UTC m=+1095.081037071"
Oct 30 12:54:11.302377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:11.302305    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr" podStartSLOduration=2.30229364 podStartE2EDuration="2.30229364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:11.299609082 +0000 UTC m=+1095.114935573" watchObservedRunningTime="2026-10-30 12:54:11.30229364 +0000 UTC m=+1095.117620126"
Oct 30 12:54:11.406123 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:11.406078    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:13.013985 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:13.013955    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:14.698676 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:14.698639    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:16.418389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:16.418340    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:17.891949 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:17.891910    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:18.127869 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:18.127744    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:18.235437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:18.232676    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:19.854672 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:19.854640    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:19.944143 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:19.944110    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:20.074652 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:20.074616    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:21.134423 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:54:21.134385    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:54:21.632080 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:21.632048    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:22.235255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:22.235227    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"
Oct 30 12:54:22.235728 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:22.235292    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"
Oct 30 12:54:23.659732 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:23.659694    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:24.239735 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:24.239703    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:25.596226 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:25.596194    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:27.368989 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:27.368950    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:28.958321 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:28.958290    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:30.256317 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:30.256283    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:30.710500 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:30.710458    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:32.356294 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:32.356259    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:34.085450 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:34.085409    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:35.838406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:35.838327    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:37.754429 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:37.751254    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:38.092868 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:38.092812    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:38.198014 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:38.197962    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:38.773194 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:38.773161    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:39.204051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.203970    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"]
Oct 30 12:54:39.209405 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.209384    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.241067 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.241034    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"]
Oct 30 12:54:39.243688 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.243666    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f9e9fd-openshift-default-dockercfg-wgwc6\""
Oct 30 12:54:39.248949 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.248887    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:39.363376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.346675    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-certs\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.363376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.346751    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-podinfo\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.363376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.346868    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-data\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.363376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.346894    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-credential-socket\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.363376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.346957    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-token\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.363376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.347016    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf92n\" (UniqueName: \"kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-kube-api-access-kf92n\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.363376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.347083    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-socket\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.363376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.347130    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istiod-ca-cert\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.363376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.347176    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-envoy\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.413866 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.413832    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:39.455808 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.455721    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-data\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.456051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.456024    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-credential-socket\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.456182 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.456167    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-token\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.456341 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.456326    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf92n\" (UniqueName: \"kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-kube-api-access-kf92n\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.456524 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.456508    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-socket\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.456643 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.456630    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istiod-ca-cert\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.456752 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.456740    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-envoy\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.456885 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.456874    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-certs\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.457003 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.456991    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-podinfo\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.472439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.458820    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-data\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.500484 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.473255    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-credential-socket\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.500718 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.474138    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istiod-ca-cert\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.500825 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.474372    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-socket\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.500898 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.480020    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-certs\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.514369 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.514310    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:39.544382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.535425    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-token\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.549377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.549260    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-podinfo\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.551451 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.551420    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-envoy\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.552460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.552437    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf92n\" (UniqueName: \"kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-kube-api-access-kf92n\") pod \"e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") " pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:39.632499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.632462    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:39.836704 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:39.836664    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:40.183446 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:54:40.183415    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608e1d64_aa04_4bc1_a4a7_244a8c02317b.slice/crio-bcd148145aa79bfa03bf2c476a52b016adba3e7ce776dae7208cc43969081075 WatchSource:0}: Error finding container bcd148145aa79bfa03bf2c476a52b016adba3e7ce776dae7208cc43969081075: Status 404 returned error can't find the container with id bcd148145aa79bfa03bf2c476a52b016adba3e7ce776dae7208cc43969081075
Oct 30 12:54:40.184681 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:40.184656    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"]
Oct 30 12:54:40.185791 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:40.185759    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:54:40.185878 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:40.185821    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:54:40.185878 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:40.185847    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:54:40.763439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:40.763402    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" event={"ID":"608e1d64-aa04-4bc1-a4a7-244a8c02317b","Type":"ContainerStarted","Data":"14a2082ba02eabb2a0c76f0b2cbf661b84877f34cdd9d92eb18b7c5a030c8408"}
Oct 30 12:54:40.763613 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:40.763488    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" event={"ID":"608e1d64-aa04-4bc1-a4a7-244a8c02317b","Type":"ContainerStarted","Data":"bcd148145aa79bfa03bf2c476a52b016adba3e7ce776dae7208cc43969081075"}
Oct 30 12:54:40.816576 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:40.816527    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" podStartSLOduration=1.8165139959999999 podStartE2EDuration="1.816513996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:40.813434043 +0000 UTC m=+1124.628760530" watchObservedRunningTime="2026-10-30 12:54:40.816513996 +0000 UTC m=+1124.631840545"
Oct 30 12:54:40.837430 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:40.837404    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:41.389808 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:41.389777    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:41.843084 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:41.843054    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:42.775179 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:42.775151    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:42.776066 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:42.776045    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:54:43.202583 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:43.202501    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:43.403930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:43.403896    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:43.537407 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:43.529875    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:45.459129 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:45.459095    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:45.557077 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:45.557046    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:46.147686 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:46.147650    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:54:46.261378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:46.251032    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:47.494223 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:47.494179    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:48.117377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:48.117010    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:49.259431 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:49.250616    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:49.319503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:49.319398    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:49.421218 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:49.421181    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:49.499881 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:49.499835    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:49.504791 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:49.504759    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:49.625527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:49.625428    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:49.977440 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:49.950637    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:54:50.075255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.075216    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:50.172377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.122457    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:50.172377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.122842    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:54:50.172763 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.172718    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:50.173249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.173220    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:50.173657 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.173633    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:54:50.584588 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.258787    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:50.585127 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.464858    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:50.585228 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.464897    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:50.585311 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.583709    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:54:50.585410 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.584084    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:50.629173 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.629100    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:50.798375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.719250    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:54:50.802376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.798648    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:54:50.903456 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.901483    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:54:50.903456 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.901547    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:54:50.974072 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:50.959449    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:51.009510 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.009480    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:54:51.128940 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.128887    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:51.136800 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.136763    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:54:51.138465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.138444    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:54:51.138654 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.138639    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:54:51.231798 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.231709    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:51.233327 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.233298    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:51.250419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.233997    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:54:51.250419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.234059    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:54:51.251546 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.251514    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 30 12:54:51.565199 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.564884    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:51.565199 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.564986    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:51.672406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.592967    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:51.672406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.593345    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:51.672406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.593634    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:54:51.672406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.593886    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:54:51.672406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.594123    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:54:51.672406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.594953    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:54:51.672406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.595015    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:54:51.672406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:51.649227    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:52.025579 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:52.025529    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:54:52.148393 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:52.147542    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:52.148393 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:52.148032    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:54:52.248559 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:52.248496    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:52.252314 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:52.252265    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:52.311302 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:52.311168    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:52.329544 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:54:52.325136    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:54:52.568434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:52.559061    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded"
Oct 30 12:54:52.593278 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:52.593225    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:52.593644 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:52.593609    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:54:52.595073 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:52.594200    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:52.595073 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:52.594547    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:54:52.595712 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:52.593816    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:52.650611 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:52.649454    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:53.036498 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:53.036447    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:53.233688 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:53.233627    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:54:53.300428 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:53.268644    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:53.300428 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:53.268978    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:53.303413 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:53.303003    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:54:53.598552 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:53.598443    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:54:53.598820 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:53.598785    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:53.599299 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:53.599270    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:54:53.615389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:53.599803    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:53.616197 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:53.601213    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:54:53.675383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:53.651369    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.054378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.053437    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.228376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.226430    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.253376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.236687    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.269483 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.269444    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.270052 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.269709    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.305517 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.305428    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.636433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.629432    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.660704 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.660647    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.681718 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.663316    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.695376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.663367    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:54:54.695376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.663429    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:54:54.695376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.663462    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.815377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.805458    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.837485 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.837440    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:54:54.837687 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.837511    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:54.981558 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:54.979055    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Oct 30 12:54:55.081378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:55.068525    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:55.257433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:55.249463    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:54:55.278092 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:55.277242    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:55.302376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:55.279962    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:55.336382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:55.309613    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:54:55.677377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:55.667252    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:55.677377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:55.673766    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:55.683382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:55.682437    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:55.708995 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:55.707987    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:54:55.710168 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:55.709231    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:55.727375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:55.711452    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:54:55.734049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:55.734017    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:56.072503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:56.072454    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:56.283531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:56.283484    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:56.383816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:56.313860    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:54:56.593736 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:56.363284    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:54:56.626086 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:56.387280    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:54:56.686480 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:56.686380    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:56.688139 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:56.686704    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:54:56.764566 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:56.686732    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:54:56.765068 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:56.733172    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:56.765698 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:56.733207    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:54:56.766383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:56.733228    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:54:56.943213 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:56.943119    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:57.078486 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.078440    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:57.240802 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.240719    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:54:57.339473 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.339432    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:54:57.446919 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.446881    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:54:57.495423 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.494871    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:54:57.519373 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.511440    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:54:57.563680 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.562175    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:54:57.563680 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.563076    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:54:57.596466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.596442    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:54:57.604870 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.603812    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:54:57.613381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.608257    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:57.643394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.638011    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:57.761526 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.760463    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:54:57.761526 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.760924    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:54:57.767877 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.767846    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:54:57.818911 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:57.818872    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:54:58.908985 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:54:58.908950    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:00.657740 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:00.657704    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:00.767122 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:00.767089    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:00.889213 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:00.885450    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:02.434957 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.434905    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:02.536337 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.536303    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:02.608222 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.608192    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"]
Oct 30 12:55:02.612592 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.612570    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"
Oct 30 12:55:02.630792 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.630770    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"]
Oct 30 12:55:02.638720 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.638701    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6bda55\""
Oct 30 12:55:02.638823 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.638769    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6bda55-dockercfg-9w7pd\""
Oct 30 12:55:02.750431 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.724785    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57r9l\" (UniqueName: \"kubernetes.io/projected/37969ed0-570c-4d9f-b726-5dbdb2596560-kube-api-access-57r9l\") pod \"payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx\" (UID: \"37969ed0-570c-4d9f-b726-5dbdb2596560\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"
Oct 30 12:55:02.750762 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.750730    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37969ed0-570c-4d9f-b726-5dbdb2596560-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx\" (UID: \"37969ed0-570c-4d9f-b726-5dbdb2596560\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"
Oct 30 12:55:02.807375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.753429    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"]
Oct 30 12:55:02.956293 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.956239    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37969ed0-570c-4d9f-b726-5dbdb2596560-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx\" (UID: \"37969ed0-570c-4d9f-b726-5dbdb2596560\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"
Oct 30 12:55:02.957174 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.957122    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57r9l\" (UniqueName: \"kubernetes.io/projected/37969ed0-570c-4d9f-b726-5dbdb2596560-kube-api-access-57r9l\") pod \"payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx\" (UID: \"37969ed0-570c-4d9f-b726-5dbdb2596560\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"
Oct 30 12:55:02.985161 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.960319    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37969ed0-570c-4d9f-b726-5dbdb2596560-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx\" (UID: \"37969ed0-570c-4d9f-b726-5dbdb2596560\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"
Oct 30 12:55:03.007707 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.999527    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"]
Oct 30 12:55:03.007707 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:02.999681    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"
Oct 30 12:55:03.019495 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:03.019463    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57r9l\" (UniqueName: \"kubernetes.io/projected/37969ed0-570c-4d9f-b726-5dbdb2596560-kube-api-access-57r9l\") pod \"payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx\" (UID: \"37969ed0-570c-4d9f-b726-5dbdb2596560\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"
Oct 30 12:55:03.084434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:03.084398    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59cc37e9-f732-42ef-89c6-638f7c1948f9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn\" (UID: \"59cc37e9-f732-42ef-89c6-638f7c1948f9\") " pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"
Oct 30 12:55:03.084877 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:03.084852    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djvnh\" (UniqueName: \"kubernetes.io/projected/59cc37e9-f732-42ef-89c6-638f7c1948f9-kube-api-access-djvnh\") pod \"payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn\" (UID: \"59cc37e9-f732-42ef-89c6-638f7c1948f9\") " pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"
Oct 30 12:55:03.165133 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:03.165105    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:03.186191 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:03.186158    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djvnh\" (UniqueName: \"kubernetes.io/projected/59cc37e9-f732-42ef-89c6-638f7c1948f9-kube-api-access-djvnh\") pod \"payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn\" (UID: \"59cc37e9-f732-42ef-89c6-638f7c1948f9\") " pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"
Oct 30 12:55:03.186314 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:03.186299    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59cc37e9-f732-42ef-89c6-638f7c1948f9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn\" (UID: \"59cc37e9-f732-42ef-89c6-638f7c1948f9\") " pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"
Oct 30 12:55:03.186960 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:03.186925    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59cc37e9-f732-42ef-89c6-638f7c1948f9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn\" (UID: \"59cc37e9-f732-42ef-89c6-638f7c1948f9\") " pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"
Oct 30 12:55:03.236114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:03.219406    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djvnh\" (UniqueName: \"kubernetes.io/projected/59cc37e9-f732-42ef-89c6-638f7c1948f9-kube-api-access-djvnh\") pod \"payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn\" (UID: \"59cc37e9-f732-42ef-89c6-638f7c1948f9\") " pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"
Oct 30 12:55:03.236114 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:03.227253    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"
Oct 30 12:55:03.363665 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:03.363591    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"
Oct 30 12:55:03.624290 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:55:03.624262    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37969ed0_570c_4d9f_b726_5dbdb2596560.slice/crio-209434b29ace7345afae7604abb6dd0b7323f6e874d1d6885b9fbf17f1a1e6fb WatchSource:0}: Error finding container 209434b29ace7345afae7604abb6dd0b7323f6e874d1d6885b9fbf17f1a1e6fb: Status 404 returned error can't find the container with id 209434b29ace7345afae7604abb6dd0b7323f6e874d1d6885b9fbf17f1a1e6fb
Oct 30 12:55:03.625920 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:03.625894    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"]
Oct 30 12:55:03.680932 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:55:03.680903    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59cc37e9_f732_42ef_89c6_638f7c1948f9.slice/crio-4f88b3d8238dd12502c67239ccd8cee1af5976b41f5c1c45ded55c4977d15c7e WatchSource:0}: Error finding container 4f88b3d8238dd12502c67239ccd8cee1af5976b41f5c1c45ded55c4977d15c7e: Status 404 returned error can't find the container with id 4f88b3d8238dd12502c67239ccd8cee1af5976b41f5c1c45ded55c4977d15c7e
Oct 30 12:55:03.681801 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:03.681766    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"]
Oct 30 12:55:04.611913 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:04.611874    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn" event={"ID":"59cc37e9-f732-42ef-89c6-638f7c1948f9","Type":"ContainerStarted","Data":"41d4c9664aff662b5e4d18ced0d61dd195883a576c507dfc4c59ffba63f427f6"}
Oct 30 12:55:04.612111 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:04.611939    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn" event={"ID":"59cc37e9-f732-42ef-89c6-638f7c1948f9","Type":"ContainerStarted","Data":"4f88b3d8238dd12502c67239ccd8cee1af5976b41f5c1c45ded55c4977d15c7e"}
Oct 30 12:55:04.612111 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:04.612086    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"
Oct 30 12:55:04.613623 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:04.613592    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx" event={"ID":"37969ed0-570c-4d9f-b726-5dbdb2596560","Type":"ContainerStarted","Data":"baebc623196d985279533c16316ddcfb9689f6f930c092223b85b500e5278833"}
Oct 30 12:55:04.613623 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:04.613626    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx" event={"ID":"37969ed0-570c-4d9f-b726-5dbdb2596560","Type":"ContainerStarted","Data":"209434b29ace7345afae7604abb6dd0b7323f6e874d1d6885b9fbf17f1a1e6fb"}
Oct 30 12:55:04.613804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:04.613725    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"
Oct 30 12:55:04.655806 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:04.655757    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn" podStartSLOduration=2.655739262 podStartE2EDuration="2.655739262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:04.653229354 +0000 UTC m=+1148.468555844" watchObservedRunningTime="2026-10-30 12:55:04.655739262 +0000 UTC m=+1148.471065754"
Oct 30 12:55:04.693818 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:04.693782    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx" podStartSLOduration=2.69377146 podStartE2EDuration="2.69377146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:04.691723844 +0000 UTC m=+1148.507050333" watchObservedRunningTime="2026-10-30 12:55:04.69377146 +0000 UTC m=+1148.509097948"
Oct 30 12:55:04.868627 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:04.868531    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:04.958257 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:04.958222    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:05.161419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:05.160452    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:06.806645 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:06.806610    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:08.600549 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:08.600515    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:10.260692 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:10.260661    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:11.993166 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:11.993126    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:13.727116 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:13.727078    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:15.213769 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:15.213734    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"]
Oct 30 12:55:15.214156 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:15.213999    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj" podUID="90570573-85f7-4227-b6fe-86abaf0804e5" containerName="istio-proxy" containerID="cri-o://48839364e34e432fbe676dc951db044ad8da72a6571b5dd806de83ff3215942b" gracePeriod=30
Oct 30 12:55:15.223476 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:15.223451    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"]
Oct 30 12:55:15.223670 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:15.223647    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt" podUID="ba4a185e-9025-45b1-bb95-c8e86a53e9bc" containerName="istio-proxy" containerID="cri-o://6488c9cbad52d5580d8d0a0a731c94f50867c340224bbb6d8e6ec5c1b1a224ad" gracePeriod=30
Oct 30 12:55:15.226622 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:15.226600    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"]
Oct 30 12:55:15.226793 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:15.226773    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h" podUID="ae991ed0-e51c-4804-b9c3-31e0152a6170" containerName="istio-proxy" containerID="cri-o://2c3c5ba36735b6e8d3dbfc93a092da964414098fd28a3dc8c5eeb6ab6759d490" gracePeriod=30
Oct 30 12:55:15.521112 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:15.521084    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:15.654836 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:15.654768    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"
Oct 30 12:55:15.661547 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:15.661515    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"
Oct 30 12:55:17.276803 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:17.276772    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:17.400623 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:17.400583    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:17.488231 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:17.486202    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:19.108279 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:19.108243    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:20.509664 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.509637    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:55:20.564051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.564025    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:55:20.568540 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.568519    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:55:20.584138 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.584110    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-socket\") pod \"90570573-85f7-4227-b6fe-86abaf0804e5\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") "
Oct 30 12:55:20.584266 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.584182    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-istio-token\" (UniqueName: \"kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-istio-token\") pod \"90570573-85f7-4227-b6fe-86abaf0804e5\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") "
Oct 30 12:55:20.584339 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.584271    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-certs\") pod \"90570573-85f7-4227-b6fe-86abaf0804e5\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") "
Oct 30 12:55:20.584417 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.584372    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-data\") pod \"90570573-85f7-4227-b6fe-86abaf0804e5\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") "
Oct 30 12:55:20.584728 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.584521    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-credential-socket\") pod \"90570573-85f7-4227-b6fe-86abaf0804e5\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") "
Oct 30 12:55:20.584728 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.584596    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90570573-85f7-4227-b6fe-86abaf0804e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90570573-85f7-4227-b6fe-86abaf0804e5-istio-podinfo\") pod \"90570573-85f7-4227-b6fe-86abaf0804e5\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") "
Oct 30 12:55:20.584728 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.584625    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-kube-api-access-swv8x\" (UniqueName: \"kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-kube-api-access-swv8x\") pod \"90570573-85f7-4227-b6fe-86abaf0804e5\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") "
Oct 30 12:55:20.584728 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.584704    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90570573-85f7-4227-b6fe-86abaf0804e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90570573-85f7-4227-b6fe-86abaf0804e5-istiod-ca-cert\") pod \"90570573-85f7-4227-b6fe-86abaf0804e5\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") "
Oct 30 12:55:20.584968 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.584746    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-envoy\") pod \"90570573-85f7-4227-b6fe-86abaf0804e5\" (UID: \"90570573-85f7-4227-b6fe-86abaf0804e5\") "
Oct 30 12:55:20.584968 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.584761    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-certs" pod "90570573-85f7-4227-b6fe-86abaf0804e5" (UID: "90570573-85f7-4227-b6fe-86abaf0804e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.586164 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.585998    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-credential-socket" pod "90570573-85f7-4227-b6fe-86abaf0804e5" (UID: "90570573-85f7-4227-b6fe-86abaf0804e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.586164 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.586057    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-data" pod "90570573-85f7-4227-b6fe-86abaf0804e5" (UID: "90570573-85f7-4227-b6fe-86abaf0804e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.586708 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.586678    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90570573-85f7-4227-b6fe-86abaf0804e5-istiod-ca-cert" pod "90570573-85f7-4227-b6fe-86abaf0804e5" (UID: "90570573-85f7-4227-b6fe-86abaf0804e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:20.587229 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.587171    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-socket" pod "90570573-85f7-4227-b6fe-86abaf0804e5" (UID: "90570573-85f7-4227-b6fe-86abaf0804e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.587883 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.587856    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.587967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.587888    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90570573-85f7-4227-b6fe-86abaf0804e5-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.587967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.587903    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.587967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.587930    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.587967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.587943    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.588698 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.588673    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-kube-api-access-swv8x" pod "90570573-85f7-4227-b6fe-86abaf0804e5" (UID: "90570573-85f7-4227-b6fe-86abaf0804e5"). InnerVolumeSpecName "kube-api-access-swv8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:20.589396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.589347    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90570573-85f7-4227-b6fe-86abaf0804e5-istio-podinfo" pod "90570573-85f7-4227-b6fe-86abaf0804e5" (UID: "90570573-85f7-4227-b6fe-86abaf0804e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:55:20.589396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.589384    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-envoy" pod "90570573-85f7-4227-b6fe-86abaf0804e5" (UID: "90570573-85f7-4227-b6fe-86abaf0804e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.590571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.590547    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-istio-token" pod "90570573-85f7-4227-b6fe-86abaf0804e5" (UID: "90570573-85f7-4227-b6fe-86abaf0804e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:20.689289 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689253    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-podinfo\") pod \"ae991ed0-e51c-4804-b9c3-31e0152a6170\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") "
Oct 30 12:55:20.689446 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689311    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-socket\") pod \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") "
Oct 30 12:55:20.689446 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689380    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-credential-socket\") pod \"ae991ed0-e51c-4804-b9c3-31e0152a6170\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") "
Oct 30 12:55:20.689446 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689424    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-envoy\") pod \"ae991ed0-e51c-4804-b9c3-31e0152a6170\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") "
Oct 30 12:55:20.689615 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689476    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istiod-ca-cert\") pod \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") "
Oct 30 12:55:20.689615 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689548    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-token\" (UniqueName: \"kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-token\") pod \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") "
Oct 30 12:55:20.689615 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689574    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-socket" pod "ba4a185e-9025-45b1-bb95-c8e86a53e9bc" (UID: "ba4a185e-9025-45b1-bb95-c8e86a53e9bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.689615 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689585    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-credential-socket\") pod \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") "
Oct 30 12:55:20.689816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689672    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-token\" (UniqueName: \"kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-token\") pod \"ae991ed0-e51c-4804-b9c3-31e0152a6170\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") "
Oct 30 12:55:20.689816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689707    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-envoy\") pod \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") "
Oct 30 12:55:20.689816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689733    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-socket\") pod \"ae991ed0-e51c-4804-b9c3-31e0152a6170\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") "
Oct 30 12:55:20.689974 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689824    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-kube-api-access-khb8b\" (UniqueName: \"kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-kube-api-access-khb8b\") pod \"ae991ed0-e51c-4804-b9c3-31e0152a6170\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") "
Oct 30 12:55:20.689974 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689850    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-credential-socket" pod "ba4a185e-9025-45b1-bb95-c8e86a53e9bc" (UID: "ba4a185e-9025-45b1-bb95-c8e86a53e9bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.689974 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689860    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-data\") pod \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") "
Oct 30 12:55:20.689974 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.689935    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istiod-ca-cert" pod "ba4a185e-9025-45b1-bb95-c8e86a53e9bc" (UID: "ba4a185e-9025-45b1-bb95-c8e86a53e9bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:20.690181 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.690012    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-certs\") pod \"ae991ed0-e51c-4804-b9c3-31e0152a6170\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") "
Oct 30 12:55:20.690181 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.690031    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-data" pod "ba4a185e-9025-45b1-bb95-c8e86a53e9bc" (UID: "ba4a185e-9025-45b1-bb95-c8e86a53e9bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.690282 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.690219    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-socket" pod "ae991ed0-e51c-4804-b9c3-31e0152a6170" (UID: "ae991ed0-e51c-4804-b9c3-31e0152a6170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.690526 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.690497    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-certs" pod "ae991ed0-e51c-4804-b9c3-31e0152a6170" (UID: "ae991ed0-e51c-4804-b9c3-31e0152a6170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.690526 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.690507    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae991ed0-e51c-4804-b9c3-31e0152a6170-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae991ed0-e51c-4804-b9c3-31e0152a6170-istiod-ca-cert\") pod \"ae991ed0-e51c-4804-b9c3-31e0152a6170\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") "
Oct 30 12:55:20.690693 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.690582    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-certs\") pod \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") "
Oct 30 12:55:20.690693 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.690664    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-data\") pod \"ae991ed0-e51c-4804-b9c3-31e0152a6170\" (UID: \"ae991ed0-e51c-4804-b9c3-31e0152a6170\") "
Oct 30 12:55:20.690797 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.690705    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-kube-api-access-psjs7\" (UniqueName: \"kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-kube-api-access-psjs7\") pod \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") "
Oct 30 12:55:20.690797 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.690787    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-podinfo\") pod \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\" (UID: \"ba4a185e-9025-45b1-bb95-c8e86a53e9bc\") "
Oct 30 12:55:20.691386 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.690786    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae991ed0-e51c-4804-b9c3-31e0152a6170-istiod-ca-cert" pod "ae991ed0-e51c-4804-b9c3-31e0152a6170" (UID: "ae991ed0-e51c-4804-b9c3-31e0152a6170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:20.691386 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.690926    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-credential-socket" pod "ae991ed0-e51c-4804-b9c3-31e0152a6170" (UID: "ae991ed0-e51c-4804-b9c3-31e0152a6170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.691550 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691120    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-data" pod "ae991ed0-e51c-4804-b9c3-31e0152a6170" (UID: "ae991ed0-e51c-4804-b9c3-31e0152a6170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.691550 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691307    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-certs" pod "ba4a185e-9025-45b1-bb95-c8e86a53e9bc" (UID: "ba4a185e-9025-45b1-bb95-c8e86a53e9bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.691656 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691640    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.691713 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691660    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.691713 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691674    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae991ed0-e51c-4804-b9c3-31e0152a6170-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.691713 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691686    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.691713 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691699    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.691713 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691711    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90570573-85f7-4227-b6fe-86abaf0804e5-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.691960 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691725    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swv8x\" (UniqueName: \"kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-kube-api-access-swv8x\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.691960 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691739    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90570573-85f7-4227-b6fe-86abaf0804e5-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.691960 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691752    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.691960 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691766    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.691960 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691780    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90570573-85f7-4227-b6fe-86abaf0804e5-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.691960 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691794    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.691960 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691806    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.691960 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.691820    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.693805 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.693781    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-kube-api-access-khb8b" pod "ae991ed0-e51c-4804-b9c3-31e0152a6170" (UID: "ae991ed0-e51c-4804-b9c3-31e0152a6170"). InnerVolumeSpecName "kube-api-access-khb8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:20.693910 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.693785    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-podinfo" pod "ae991ed0-e51c-4804-b9c3-31e0152a6170" (UID: "ae991ed0-e51c-4804-b9c3-31e0152a6170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:55:20.693910 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.693883    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-envoy" pod "ae991ed0-e51c-4804-b9c3-31e0152a6170" (UID: "ae991ed0-e51c-4804-b9c3-31e0152a6170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.694021 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.693990    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-token" pod "ae991ed0-e51c-4804-b9c3-31e0152a6170" (UID: "ae991ed0-e51c-4804-b9c3-31e0152a6170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:20.694342 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.694313    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-envoy" pod "ba4a185e-9025-45b1-bb95-c8e86a53e9bc" (UID: "ba4a185e-9025-45b1-bb95-c8e86a53e9bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:20.694905 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.694872    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-token" pod "ba4a185e-9025-45b1-bb95-c8e86a53e9bc" (UID: "ba4a185e-9025-45b1-bb95-c8e86a53e9bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:20.695718 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.695699    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-podinfo" pod "ba4a185e-9025-45b1-bb95-c8e86a53e9bc" (UID: "ba4a185e-9025-45b1-bb95-c8e86a53e9bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:55:20.696246 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.696221    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-kube-api-access-psjs7" pod "ba4a185e-9025-45b1-bb95-c8e86a53e9bc" (UID: "ba4a185e-9025-45b1-bb95-c8e86a53e9bc"). InnerVolumeSpecName "kube-api-access-psjs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:20.793500 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.793474    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.793500 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.793500    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.793662 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.793515    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.793662 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.793526    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.793662 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.793534    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khb8b\" (UniqueName: \"kubernetes.io/projected/ae991ed0-e51c-4804-b9c3-31e0152a6170-kube-api-access-khb8b\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.793662 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.793544    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psjs7\" (UniqueName: \"kubernetes.io/projected/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-kube-api-access-psjs7\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.793662 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.793552    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4a185e-9025-45b1-bb95-c8e86a53e9bc-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.793662 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.793560    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae991ed0-e51c-4804-b9c3-31e0152a6170-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:20.794418 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.794389    4522 generic.go:356] "Generic (PLEG): container finished" podID="90570573-85f7-4227-b6fe-86abaf0804e5" containerID="48839364e34e432fbe676dc951db044ad8da72a6571b5dd806de83ff3215942b" exitCode=0
Oct 30 12:55:20.794515 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.794437    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"
Oct 30 12:55:20.794515 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.794479    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj" event={"ID":"90570573-85f7-4227-b6fe-86abaf0804e5","Type":"ContainerDied","Data":"48839364e34e432fbe676dc951db044ad8da72a6571b5dd806de83ff3215942b"}
Oct 30 12:55:20.794633 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.794532    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj" event={"ID":"90570573-85f7-4227-b6fe-86abaf0804e5","Type":"ContainerDied","Data":"669aef18a2416057cb82c7bf7e615b10b963077f956de554b3cb44de8f3feb90"}
Oct 30 12:55:20.794633 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.794554    4522 scope.go:122] "RemoveContainer" containerID="48839364e34e432fbe676dc951db044ad8da72a6571b5dd806de83ff3215942b"
Oct 30 12:55:20.796106 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.796079    4522 generic.go:356] "Generic (PLEG): container finished" podID="ba4a185e-9025-45b1-bb95-c8e86a53e9bc" containerID="6488c9cbad52d5580d8d0a0a731c94f50867c340224bbb6d8e6ec5c1b1a224ad" exitCode=0
Oct 30 12:55:20.796214 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.796127    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"
Oct 30 12:55:20.796214 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.796177    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt" event={"ID":"ba4a185e-9025-45b1-bb95-c8e86a53e9bc","Type":"ContainerDied","Data":"6488c9cbad52d5580d8d0a0a731c94f50867c340224bbb6d8e6ec5c1b1a224ad"}
Oct 30 12:55:20.796308 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.796228    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt" event={"ID":"ba4a185e-9025-45b1-bb95-c8e86a53e9bc","Type":"ContainerDied","Data":"79b2b72e37c7a3c9d212263c70a5291c8d7876ad97a6c51a4b9d1114b0f04396"}
Oct 30 12:55:20.797536 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.797516    4522 generic.go:356] "Generic (PLEG): container finished" podID="ae991ed0-e51c-4804-b9c3-31e0152a6170" containerID="2c3c5ba36735b6e8d3dbfc93a092da964414098fd28a3dc8c5eeb6ab6759d490" exitCode=0
Oct 30 12:55:20.797639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.797554    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"
Oct 30 12:55:20.797639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.797606    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h" event={"ID":"ae991ed0-e51c-4804-b9c3-31e0152a6170","Type":"ContainerDied","Data":"2c3c5ba36735b6e8d3dbfc93a092da964414098fd28a3dc8c5eeb6ab6759d490"}
Oct 30 12:55:20.797752 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.797642    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h" event={"ID":"ae991ed0-e51c-4804-b9c3-31e0152a6170","Type":"ContainerDied","Data":"27beb08316d108514dcde220c16cc4534eeed13df54ce1270f272b35922acc0f"}
Oct 30 12:55:20.806288 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.806261    4522 scope.go:122] "RemoveContainer" containerID="48839364e34e432fbe676dc951db044ad8da72a6571b5dd806de83ff3215942b"
Oct 30 12:55:20.808320 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:55:20.807717    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48839364e34e432fbe676dc951db044ad8da72a6571b5dd806de83ff3215942b\": container with ID starting with 48839364e34e432fbe676dc951db044ad8da72a6571b5dd806de83ff3215942b not found: ID does not exist" containerID="48839364e34e432fbe676dc951db044ad8da72a6571b5dd806de83ff3215942b"
Oct 30 12:55:20.808320 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.807764    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48839364e34e432fbe676dc951db044ad8da72a6571b5dd806de83ff3215942b"} err="failed to get container status \"48839364e34e432fbe676dc951db044ad8da72a6571b5dd806de83ff3215942b\": rpc error: code = NotFound desc = could not find container \"48839364e34e432fbe676dc951db044ad8da72a6571b5dd806de83ff3215942b\": container with ID starting with 48839364e34e432fbe676dc951db044ad8da72a6571b5dd806de83ff3215942b not found: ID does not exist"
Oct 30 12:55:20.808320 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.807788    4522 scope.go:122] "RemoveContainer" containerID="6488c9cbad52d5580d8d0a0a731c94f50867c340224bbb6d8e6ec5c1b1a224ad"
Oct 30 12:55:20.811524 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.811500    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:20.825365 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.825336    4522 scope.go:122] "RemoveContainer" containerID="6488c9cbad52d5580d8d0a0a731c94f50867c340224bbb6d8e6ec5c1b1a224ad"
Oct 30 12:55:20.825630 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:55:20.825611    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6488c9cbad52d5580d8d0a0a731c94f50867c340224bbb6d8e6ec5c1b1a224ad\": container with ID starting with 6488c9cbad52d5580d8d0a0a731c94f50867c340224bbb6d8e6ec5c1b1a224ad not found: ID does not exist" containerID="6488c9cbad52d5580d8d0a0a731c94f50867c340224bbb6d8e6ec5c1b1a224ad"
Oct 30 12:55:20.825710 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.825640    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6488c9cbad52d5580d8d0a0a731c94f50867c340224bbb6d8e6ec5c1b1a224ad"} err="failed to get container status \"6488c9cbad52d5580d8d0a0a731c94f50867c340224bbb6d8e6ec5c1b1a224ad\": rpc error: code = NotFound desc = could not find container \"6488c9cbad52d5580d8d0a0a731c94f50867c340224bbb6d8e6ec5c1b1a224ad\": container with ID starting with 6488c9cbad52d5580d8d0a0a731c94f50867c340224bbb6d8e6ec5c1b1a224ad not found: ID does not exist"
Oct 30 12:55:20.825710 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.825660    4522 scope.go:122] "RemoveContainer" containerID="2c3c5ba36735b6e8d3dbfc93a092da964414098fd28a3dc8c5eeb6ab6759d490"
Oct 30 12:55:20.834256 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.834240    4522 scope.go:122] "RemoveContainer" containerID="2c3c5ba36735b6e8d3dbfc93a092da964414098fd28a3dc8c5eeb6ab6759d490"
Oct 30 12:55:20.834558 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:55:20.834540    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3c5ba36735b6e8d3dbfc93a092da964414098fd28a3dc8c5eeb6ab6759d490\": container with ID starting with 2c3c5ba36735b6e8d3dbfc93a092da964414098fd28a3dc8c5eeb6ab6759d490 not found: ID does not exist" containerID="2c3c5ba36735b6e8d3dbfc93a092da964414098fd28a3dc8c5eeb6ab6759d490"
Oct 30 12:55:20.834618 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.834563    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3c5ba36735b6e8d3dbfc93a092da964414098fd28a3dc8c5eeb6ab6759d490"} err="failed to get container status \"2c3c5ba36735b6e8d3dbfc93a092da964414098fd28a3dc8c5eeb6ab6759d490\": rpc error: code = NotFound desc = could not find container \"2c3c5ba36735b6e8d3dbfc93a092da964414098fd28a3dc8c5eeb6ab6759d490\": container with ID starting with 2c3c5ba36735b6e8d3dbfc93a092da964414098fd28a3dc8c5eeb6ab6759d490 not found: ID does not exist"
Oct 30 12:55:20.916283 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.916227    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"]
Oct 30 12:55:20.922523 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.922472    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8d5wt"]
Oct 30 12:55:20.956773 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.945562    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"]
Oct 30 12:55:20.956773 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.950100    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j2w5h"]
Oct 30 12:55:20.983491 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.983424    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"]
Oct 30 12:55:20.986595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:20.986568    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nllbj"]
Oct 30 12:55:21.133882 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:55:21.133814    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:55:22.334101 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:22.334068    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90570573-85f7-4227-b6fe-86abaf0804e5" path="/var/lib/kubelet/pods/90570573-85f7-4227-b6fe-86abaf0804e5/volumes"
Oct 30 12:55:22.334706 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:22.334685    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae991ed0-e51c-4804-b9c3-31e0152a6170" path="/var/lib/kubelet/pods/ae991ed0-e51c-4804-b9c3-31e0152a6170/volumes"
Oct 30 12:55:22.335202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:22.335181    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4a185e-9025-45b1-bb95-c8e86a53e9bc" path="/var/lib/kubelet/pods/ba4a185e-9025-45b1-bb95-c8e86a53e9bc/volumes"
Oct 30 12:55:22.492817 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:22.492789    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:24.178003 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:24.177967    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:25.905915 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:25.905878    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:27.619807 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:27.619770    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:28.852083 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:28.852050    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:29.352534 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:29.352501    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:31.116676 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:31.116642    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:32.828714 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:32.828678    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:34.597085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:34.597040    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:36.366103 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:36.366070    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:38.192655 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:38.192620    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:39.908191 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:39.908144    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:41.109201 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:41.109170    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:41.598651 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:41.598604    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:41.743560 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:41.743461    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:41.858239 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:41.858163    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"]
Oct 30 12:55:41.893839 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:41.893808    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"]
Oct 30 12:55:41.986367 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:41.986330    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:42.086882 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.086850    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:42.161055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.160982    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"]
Oct 30 12:55:42.161905 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.161880    4522 cpu_manager.go:414] "containerMap: removing container" podUID="ae991ed0-e51c-4804-b9c3-31e0152a6170" containerName="istio-proxy"
Oct 30 12:55:42.161905 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.161906    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae991ed0-e51c-4804-b9c3-31e0152a6170" containerName="istio-proxy"
Oct 30 12:55:42.162088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.161927    4522 cpu_manager.go:414] "containerMap: removing container" podUID="90570573-85f7-4227-b6fe-86abaf0804e5" containerName="istio-proxy"
Oct 30 12:55:42.162088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.161935    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90570573-85f7-4227-b6fe-86abaf0804e5" containerName="istio-proxy"
Oct 30 12:55:42.162088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.161971    4522 cpu_manager.go:414] "containerMap: removing container" podUID="ba4a185e-9025-45b1-bb95-c8e86a53e9bc" containerName="istio-proxy"
Oct 30 12:55:42.162088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.162008    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4a185e-9025-45b1-bb95-c8e86a53e9bc" containerName="istio-proxy"
Oct 30 12:55:42.162295 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.162158    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae991ed0-e51c-4804-b9c3-31e0152a6170" containerName="istio-proxy"
Oct 30 12:55:42.162295 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.162173    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="90570573-85f7-4227-b6fe-86abaf0804e5" containerName="istio-proxy"
Oct 30 12:55:42.162295 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.162185    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4a185e-9025-45b1-bb95-c8e86a53e9bc" containerName="istio-proxy"
Oct 30 12:55:42.168088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.168069    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.184577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.184551    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"]
Oct 30 12:55:42.184789 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.184771    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6a4499-openshift-default-dockercfg-2jtjs\""
Oct 30 12:55:42.203404 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.203375    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:42.256249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.256218    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-podinfo\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.256395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.256260    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-socket\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.256479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.256458    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daea3195-b3c7-4500-9dfe-b753efa60e0e-istiod-ca-cert\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.256549 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.256517    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-certs\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.256605 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.256552    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-credential-socket\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.256683 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.256665    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-token\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.256732 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.256720    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-envoy\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.256772 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.256762    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-data\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.256869 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.256842    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24q9j\" (UniqueName: \"kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-kube-api-access-24q9j\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.304550 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.304529    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:42.358135 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.358111    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-token\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.358225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.358159    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-envoy\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.358225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.358196    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-data\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.358225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.358219    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24q9j\" (UniqueName: \"kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-kube-api-access-24q9j\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.358339 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.358320    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-podinfo\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.358409 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.358387    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-socket\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.358472 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.358456    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daea3195-b3c7-4500-9dfe-b753efa60e0e-istiod-ca-cert\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.358519 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.358492    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-certs\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.358577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.358523    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-credential-socket\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.358682 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.358665    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-data\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.358799 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.358780    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-socket\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.358856 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.358797    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-credential-socket\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.358924 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.358896    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-certs\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.359318 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.359293    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daea3195-b3c7-4500-9dfe-b753efa60e0e-istiod-ca-cert\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.360553 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.360535    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-envoy\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.360779 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.360762    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-podinfo\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.376890 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.376867    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24q9j\" (UniqueName: \"kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-kube-api-access-24q9j\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.377324 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.377309    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-token\") pod \"e2e-api-b-6a4499-openshift-default-8677876f85-2m44b\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") " pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:42.495489 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:42.495233    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:43.036897 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:55:43.036863    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaea3195_b3c7_4500_9dfe_b753efa60e0e.slice/crio-0eff509ccebf39580298e3c4ee9e472c24bf4aa9cfc25eab29e65ead951939fe WatchSource:0}: Error finding container 0eff509ccebf39580298e3c4ee9e472c24bf4aa9cfc25eab29e65ead951939fe: Status 404 returned error can't find the container with id 0eff509ccebf39580298e3c4ee9e472c24bf4aa9cfc25eab29e65ead951939fe
Oct 30 12:55:43.037816 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:43.037789    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"]
Oct 30 12:55:43.039348 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:43.039314    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:55:43.039454 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:43.039393    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:55:43.039454 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:43.039423    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:55:43.401081 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:43.401003    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:43.960011 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:43.959975    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" event={"ID":"daea3195-b3c7-4500-9dfe-b753efa60e0e","Type":"ContainerStarted","Data":"2533e2550f8eb41c5a66db88615c8b11fca3bc7eae04269b6c821f1f56c22673"}
Oct 30 12:55:43.960186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:43.960041    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" event={"ID":"daea3195-b3c7-4500-9dfe-b753efa60e0e","Type":"ContainerStarted","Data":"0eff509ccebf39580298e3c4ee9e472c24bf4aa9cfc25eab29e65ead951939fe"}
Oct 30 12:55:44.013797 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:44.013752    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" podStartSLOduration=2.013740441 podStartE2EDuration="2.013740441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:44.011110186 +0000 UTC m=+1187.826436677" watchObservedRunningTime="2026-10-30 12:55:44.013740441 +0000 UTC m=+1187.829066922"
Oct 30 12:55:44.496403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:44.496376    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:44.501592 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:44.501569    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:44.966291 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:44.966265    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:44.967065 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:44.967047    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:55:45.210579 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:45.210544    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:46.343613 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:46.343577    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:46.445469 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:46.445437    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:46.858909 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:46.858849    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr" podUID="d2a8150a-f0de-4f31-94de-d44575f13f05" containerName="payload-processing" containerID="cri-o://ab088d07ea34bd23e903017c2024a7a4915457f8cb3590b4a7faf6ea2fecf7f9" gracePeriod=25
Oct 30 12:55:46.938854 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:46.938827    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9" podUID="64b94153-ad76-4ce3-8afd-a05398816dd4" containerName="payload-pre-processing" containerID="cri-o://05e4fc573f6062e033ebbb760f21f076ae148f53bb82712bfa7ccdd58ad25814" gracePeriod=25
Oct 30 12:55:46.979869 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:46.979845    4522 generic.go:356] "Generic (PLEG): container finished" podID="d2a8150a-f0de-4f31-94de-d44575f13f05" containerID="ab088d07ea34bd23e903017c2024a7a4915457f8cb3590b4a7faf6ea2fecf7f9" exitCode=0
Oct 30 12:55:46.979966 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:46.979916    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr" event={"ID":"d2a8150a-f0de-4f31-94de-d44575f13f05","Type":"ContainerDied","Data":"ab088d07ea34bd23e903017c2024a7a4915457f8cb3590b4a7faf6ea2fecf7f9"}
Oct 30 12:55:47.218456 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.218425    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:47.225753 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.225723    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"
Oct 30 12:55:47.255167 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.255146    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"
Oct 30 12:55:47.338374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.331972    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a8150a-f0de-4f31-94de-d44575f13f05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a8150a-f0de-4f31-94de-d44575f13f05-plugins-config-volume\") pod \"d2a8150a-f0de-4f31-94de-d44575f13f05\" (UID: \"d2a8150a-f0de-4f31-94de-d44575f13f05\") "
Oct 30 12:55:47.338374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.332123    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b94153-ad76-4ce3-8afd-a05398816dd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b94153-ad76-4ce3-8afd-a05398816dd4-plugins-config-volume\") pod \"64b94153-ad76-4ce3-8afd-a05398816dd4\" (UID: \"64b94153-ad76-4ce3-8afd-a05398816dd4\") "
Oct 30 12:55:47.338374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.332255    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a8150a-f0de-4f31-94de-d44575f13f05-kube-api-access-gjljs\" (UniqueName: \"kubernetes.io/projected/d2a8150a-f0de-4f31-94de-d44575f13f05-kube-api-access-gjljs\") pod \"d2a8150a-f0de-4f31-94de-d44575f13f05\" (UID: \"d2a8150a-f0de-4f31-94de-d44575f13f05\") "
Oct 30 12:55:47.338374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.332300    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b94153-ad76-4ce3-8afd-a05398816dd4-kube-api-access-52ml4\" (UniqueName: \"kubernetes.io/projected/64b94153-ad76-4ce3-8afd-a05398816dd4-kube-api-access-52ml4\") pod \"64b94153-ad76-4ce3-8afd-a05398816dd4\" (UID: \"64b94153-ad76-4ce3-8afd-a05398816dd4\") "
Oct 30 12:55:47.363731 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.363171    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a8150a-f0de-4f31-94de-d44575f13f05-plugins-config-volume" pod "d2a8150a-f0de-4f31-94de-d44575f13f05" (UID: "d2a8150a-f0de-4f31-94de-d44575f13f05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:47.363731 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.363677    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b94153-ad76-4ce3-8afd-a05398816dd4-plugins-config-volume" pod "64b94153-ad76-4ce3-8afd-a05398816dd4" (UID: "64b94153-ad76-4ce3-8afd-a05398816dd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:47.388329 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.388248    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a8150a-f0de-4f31-94de-d44575f13f05-kube-api-access-gjljs" pod "d2a8150a-f0de-4f31-94de-d44575f13f05" (UID: "d2a8150a-f0de-4f31-94de-d44575f13f05"). InnerVolumeSpecName "kube-api-access-gjljs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:47.388772 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.388719    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b94153-ad76-4ce3-8afd-a05398816dd4-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:47.388772 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.388748    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a8150a-f0de-4f31-94de-d44575f13f05-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:47.390695 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.390633    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b94153-ad76-4ce3-8afd-a05398816dd4-kube-api-access-52ml4" pod "64b94153-ad76-4ce3-8afd-a05398816dd4" (UID: "64b94153-ad76-4ce3-8afd-a05398816dd4"). InnerVolumeSpecName "kube-api-access-52ml4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:47.489934 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.489846    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjljs\" (UniqueName: \"kubernetes.io/projected/d2a8150a-f0de-4f31-94de-d44575f13f05-kube-api-access-gjljs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:47.489934 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.489880    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52ml4\" (UniqueName: \"kubernetes.io/projected/64b94153-ad76-4ce3-8afd-a05398816dd4-kube-api-access-52ml4\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:55:47.987777 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.987738    4522 generic.go:356] "Generic (PLEG): container finished" podID="64b94153-ad76-4ce3-8afd-a05398816dd4" containerID="05e4fc573f6062e033ebbb760f21f076ae148f53bb82712bfa7ccdd58ad25814" exitCode=0
Oct 30 12:55:47.987930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.987781    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"
Oct 30 12:55:47.987930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.987775    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9" event={"ID":"64b94153-ad76-4ce3-8afd-a05398816dd4","Type":"ContainerDied","Data":"05e4fc573f6062e033ebbb760f21f076ae148f53bb82712bfa7ccdd58ad25814"}
Oct 30 12:55:47.987930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.987913    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9" event={"ID":"64b94153-ad76-4ce3-8afd-a05398816dd4","Type":"ContainerDied","Data":"5751afc617818f4400b01ec4916f26060365bcc0f84f63b10632addfee2b3b61"}
Oct 30 12:55:47.988100 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.987947    4522 scope.go:122] "RemoveContainer" containerID="05e4fc573f6062e033ebbb760f21f076ae148f53bb82712bfa7ccdd58ad25814"
Oct 30 12:55:47.989328 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.989311    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"
Oct 30 12:55:47.989493 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:47.989326    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr" event={"ID":"d2a8150a-f0de-4f31-94de-d44575f13f05","Type":"ContainerDied","Data":"d663315833fce049eddc16b56c5bda3f0dff849c7972f53e252aadfd9ffd6a3c"}
Oct 30 12:55:48.001731 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.001711    4522 scope.go:122] "RemoveContainer" containerID="05e4fc573f6062e033ebbb760f21f076ae148f53bb82712bfa7ccdd58ad25814"
Oct 30 12:55:48.002500 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:55:48.002479    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e4fc573f6062e033ebbb760f21f076ae148f53bb82712bfa7ccdd58ad25814\": container with ID starting with 05e4fc573f6062e033ebbb760f21f076ae148f53bb82712bfa7ccdd58ad25814 not found: ID does not exist" containerID="05e4fc573f6062e033ebbb760f21f076ae148f53bb82712bfa7ccdd58ad25814"
Oct 30 12:55:48.002599 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.002506    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e4fc573f6062e033ebbb760f21f076ae148f53bb82712bfa7ccdd58ad25814"} err="failed to get container status \"05e4fc573f6062e033ebbb760f21f076ae148f53bb82712bfa7ccdd58ad25814\": rpc error: code = NotFound desc = could not find container \"05e4fc573f6062e033ebbb760f21f076ae148f53bb82712bfa7ccdd58ad25814\": container with ID starting with 05e4fc573f6062e033ebbb760f21f076ae148f53bb82712bfa7ccdd58ad25814 not found: ID does not exist"
Oct 30 12:55:48.002599 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.002532    4522 scope.go:122] "RemoveContainer" containerID="ab088d07ea34bd23e903017c2024a7a4915457f8cb3590b4a7faf6ea2fecf7f9"
Oct 30 12:55:48.003376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.003339    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:48.043052 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.043022    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"]
Oct 30 12:55:48.054606 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.054583    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a531c-78d5c4c9c9-58xs9"]
Oct 30 12:55:48.086476 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.086456    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"]
Oct 30 12:55:48.089157 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.089132    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:48.090187 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.090169    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a531c-dc7fd5988-k4svr"]
Oct 30 12:55:48.123603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.123550    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"]
Oct 30 12:55:48.124064 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.124052    4522 cpu_manager.go:414] "containerMap: removing container" podUID="d2a8150a-f0de-4f31-94de-d44575f13f05" containerName="payload-processing"
Oct 30 12:55:48.124100 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.124066    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a8150a-f0de-4f31-94de-d44575f13f05" containerName="payload-processing"
Oct 30 12:55:48.124100 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.124096    4522 cpu_manager.go:414] "containerMap: removing container" podUID="64b94153-ad76-4ce3-8afd-a05398816dd4" containerName="payload-pre-processing"
Oct 30 12:55:48.124177 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.124101    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b94153-ad76-4ce3-8afd-a05398816dd4" containerName="payload-pre-processing"
Oct 30 12:55:48.124219 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.124210    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b94153-ad76-4ce3-8afd-a05398816dd4" containerName="payload-pre-processing"
Oct 30 12:55:48.124249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.124224    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a8150a-f0de-4f31-94de-d44575f13f05" containerName="payload-processing"
Oct 30 12:55:48.129046 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.129029    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"
Oct 30 12:55:48.144091 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.144070    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6a4499\""
Oct 30 12:55:48.147078 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.147058    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"]
Oct 30 12:55:48.191932 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.191904    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"]
Oct 30 12:55:48.196848 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.196831    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"
Oct 30 12:55:48.197631 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.197609    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rph9x\" (UniqueName: \"kubernetes.io/projected/7237c611-57be-484a-b096-0e12164ab8eb-kube-api-access-rph9x\") pod \"payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk\" (UID: \"7237c611-57be-484a-b096-0e12164ab8eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"
Oct 30 12:55:48.197723 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.197704    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7237c611-57be-484a-b096-0e12164ab8eb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk\" (UID: \"7237c611-57be-484a-b096-0e12164ab8eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"
Oct 30 12:55:48.210564 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.210544    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"]
Oct 30 12:55:48.212013 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.211983    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:48.265379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.245741    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6a4499-dockercfg-tvqp2\""
Oct 30 12:55:48.321376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.315721    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7237c611-57be-484a-b096-0e12164ab8eb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk\" (UID: \"7237c611-57be-484a-b096-0e12164ab8eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"
Oct 30 12:55:48.321376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.315803    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w277s\" (UniqueName: \"kubernetes.io/projected/8e969a4e-090b-4931-a158-8b45c1c29eb2-kube-api-access-w277s\") pod \"payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59\" (UID: \"8e969a4e-090b-4931-a158-8b45c1c29eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"
Oct 30 12:55:48.321376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.315842    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e969a4e-090b-4931-a158-8b45c1c29eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59\" (UID: \"8e969a4e-090b-4931-a158-8b45c1c29eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"
Oct 30 12:55:48.321376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.316018    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rph9x\" (UniqueName: \"kubernetes.io/projected/7237c611-57be-484a-b096-0e12164ab8eb-kube-api-access-rph9x\") pod \"payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk\" (UID: \"7237c611-57be-484a-b096-0e12164ab8eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"
Oct 30 12:55:48.321376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.317185    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7237c611-57be-484a-b096-0e12164ab8eb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk\" (UID: \"7237c611-57be-484a-b096-0e12164ab8eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"
Oct 30 12:55:48.440428 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.440189    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w277s\" (UniqueName: \"kubernetes.io/projected/8e969a4e-090b-4931-a158-8b45c1c29eb2-kube-api-access-w277s\") pod \"payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59\" (UID: \"8e969a4e-090b-4931-a158-8b45c1c29eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"
Oct 30 12:55:48.440428 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.440258    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e969a4e-090b-4931-a158-8b45c1c29eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59\" (UID: \"8e969a4e-090b-4931-a158-8b45c1c29eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"
Oct 30 12:55:48.446881 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.441545    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e969a4e-090b-4931-a158-8b45c1c29eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59\" (UID: \"8e969a4e-090b-4931-a158-8b45c1c29eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"
Oct 30 12:55:48.535382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.479829    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rph9x\" (UniqueName: \"kubernetes.io/projected/7237c611-57be-484a-b096-0e12164ab8eb-kube-api-access-rph9x\") pod \"payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk\" (UID: \"7237c611-57be-484a-b096-0e12164ab8eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"
Oct 30 12:55:48.535382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.485814    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"
Oct 30 12:55:48.535382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.504285    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b94153-ad76-4ce3-8afd-a05398816dd4" path="/var/lib/kubelet/pods/64b94153-ad76-4ce3-8afd-a05398816dd4/volumes"
Oct 30 12:55:48.535382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.505005    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a8150a-f0de-4f31-94de-d44575f13f05" path="/var/lib/kubelet/pods/d2a8150a-f0de-4f31-94de-d44575f13f05/volumes"
Oct 30 12:55:48.545578 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.545399    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w277s\" (UniqueName: \"kubernetes.io/projected/8e969a4e-090b-4931-a158-8b45c1c29eb2-kube-api-access-w277s\") pod \"payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59\" (UID: \"8e969a4e-090b-4931-a158-8b45c1c29eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"
Oct 30 12:55:48.811428 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.811404    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"
Oct 30 12:55:48.844342 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:55:48.844316    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7237c611_57be_484a_b096_0e12164ab8eb.slice/crio-116a777b51f2736b18addcb0b3c908a28a25970e144b1aed695869920e6a6107 WatchSource:0}: Error finding container 116a777b51f2736b18addcb0b3c908a28a25970e144b1aed695869920e6a6107: Status 404 returned error can't find the container with id 116a777b51f2736b18addcb0b3c908a28a25970e144b1aed695869920e6a6107
Oct 30 12:55:48.846093 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.846016    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"]
Oct 30 12:55:48.997898 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.997870    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk" event={"ID":"7237c611-57be-484a-b096-0e12164ab8eb","Type":"ContainerStarted","Data":"cf3bbe7d668e5b5aa27a6ce6e71f95e57da90445d306c902228010a4706882bf"}
Oct 30 12:55:48.998022 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.997915    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk" event={"ID":"7237c611-57be-484a-b096-0e12164ab8eb","Type":"ContainerStarted","Data":"116a777b51f2736b18addcb0b3c908a28a25970e144b1aed695869920e6a6107"}
Oct 30 12:55:48.998022 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:48.998006    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"
Oct 30 12:55:49.012741 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:49.012706    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:49.040566 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:49.040521    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk" podStartSLOduration=1.040508767 podStartE2EDuration="1.040508767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:49.038424177 +0000 UTC m=+1192.853750678" watchObservedRunningTime="2026-10-30 12:55:49.040508767 +0000 UTC m=+1192.855835255"
Oct 30 12:55:49.113520 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:55:49.113489    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e969a4e_090b_4931_a158_8b45c1c29eb2.slice/crio-53b92e78be2a5a3d3deb79bdb60821c98170cdfca8be7c4a37c10ef91d470fc3 WatchSource:0}: Error finding container 53b92e78be2a5a3d3deb79bdb60821c98170cdfca8be7c4a37c10ef91d470fc3: Status 404 returned error can't find the container with id 53b92e78be2a5a3d3deb79bdb60821c98170cdfca8be7c4a37c10ef91d470fc3
Oct 30 12:55:49.116627 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:49.115206    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"]
Oct 30 12:55:50.005275 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:50.005243    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59" event={"ID":"8e969a4e-090b-4931-a158-8b45c1c29eb2","Type":"ContainerStarted","Data":"c475d060e0c500fe804dd953641ac2b9bc1afc3f7e92108bb7a7ea6517ed4885"}
Oct 30 12:55:50.005672 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:50.005295    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59" event={"ID":"8e969a4e-090b-4931-a158-8b45c1c29eb2","Type":"ContainerStarted","Data":"53b92e78be2a5a3d3deb79bdb60821c98170cdfca8be7c4a37c10ef91d470fc3"}
Oct 30 12:55:50.005672 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:50.005335    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"
Oct 30 12:55:50.047095 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:50.047044    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59" podStartSLOduration=2.047028371 podStartE2EDuration="2.047028371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:50.041943408 +0000 UTC m=+1193.857269899" watchObservedRunningTime="2026-10-30 12:55:50.047028371 +0000 UTC m=+1193.862354861"
Oct 30 12:55:50.681203 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:50.681170    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:51.133280 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:55:51.133246    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:55:52.316249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:52.316214    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:53.989050 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:53.989020    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:55.676958 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:55.676921    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:57.298882 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:57.298824    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:58.991918 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:55:58.991883    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:00.008021 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.007994    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"
Oct 30 12:56:00.573618 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.573582    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"]
Oct 30 12:56:00.579292 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.579270    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:00.598479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.598455    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"]
Oct 30 12:56:00.615366 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.605995    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6a4499-serving-cert\""
Oct 30 12:56:00.653236 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.653208    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:00.735099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.735074    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r282\" (UniqueName: \"kubernetes.io/projected/c566749d-af52-4992-803c-334d8e0d08c6-kube-api-access-4r282\") pod \"maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:00.735209 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.735110    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c566749d-af52-4992-803c-334d8e0d08c6-openshift-service-ca\") pod \"maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:00.735209 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.735199    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-maas-api-tls\") pod \"maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:00.735290 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.735235    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-metrics-certs\") pod \"maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:00.744871 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.744850    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:00.836018 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.835953    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-metrics-certs\") pod \"maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:00.836142 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.836121    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r282\" (UniqueName: \"kubernetes.io/projected/c566749d-af52-4992-803c-334d8e0d08c6-kube-api-access-4r282\") pod \"maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:00.836223 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.836177    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c566749d-af52-4992-803c-334d8e0d08c6-openshift-service-ca\") pod \"maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:00.836288 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.836270    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-maas-api-tls\") pod \"maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:00.836765 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.836742    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c566749d-af52-4992-803c-334d8e0d08c6-openshift-service-ca\") pod \"maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:00.838330 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.838309    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-metrics-certs\") pod \"maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:00.838561 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.838544    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-maas-api-tls\") pod \"maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:00.845338 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.845304    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:00.918375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.914208    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r282\" (UniqueName: \"kubernetes.io/projected/c566749d-af52-4992-803c-334d8e0d08c6-kube-api-access-4r282\") pod \"maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:00.918375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:00.914539    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:01.021042 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:01.018675    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"
Oct 30 12:56:01.313606 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:56:01.313578    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc566749d_af52_4992_803c_334d8e0d08c6.slice/crio-4f0ccb08c7ed196c0bea68124b705982f2484fc8f23c4445a928537e5004efc8 WatchSource:0}: Error finding container 4f0ccb08c7ed196c0bea68124b705982f2484fc8f23c4445a928537e5004efc8: Status 404 returned error can't find the container with id 4f0ccb08c7ed196c0bea68124b705982f2484fc8f23c4445a928537e5004efc8
Oct 30 12:56:01.314533 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:01.314505    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"]
Oct 30 12:56:01.315869 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:01.315851    4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:56:02.174053 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:02.174017    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9" event={"ID":"c566749d-af52-4992-803c-334d8e0d08c6","Type":"ContainerStarted","Data":"6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287"}
Oct 30 12:56:02.174471 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:02.174073    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9" event={"ID":"c566749d-af52-4992-803c-334d8e0d08c6","Type":"ContainerStarted","Data":"4f0ccb08c7ed196c0bea68124b705982f2484fc8f23c4445a928537e5004efc8"}
Oct 30 12:56:02.174471 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:02.174132    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:02.216142 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:02.216098    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9" podStartSLOduration=1.842935048 podStartE2EDuration="2.216085201s" totalImagesPullingTime="373.150153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:02.21454083 +0000 UTC m=+1206.029867317" watchObservedRunningTime="2026-10-30 12:56:02.216085201 +0000 UTC m=+1206.031411732"
Oct 30 12:56:02.445689 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:02.445607    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:02.666259 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:02.666213    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:56:04.378611 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:04.378573    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:06.307388 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:06.306456    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:06.609166 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:06.608884    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:06.662385 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:06.649622    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:06.662385 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:06.650844    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:56:06.666727 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:06.666689    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:06.695373 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:06.689639    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:56:06.695373 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:06.690605    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:56:06.830104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:06.830057    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:56:06.855503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:06.855453    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:56:07.048433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.027413    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:56:07.048433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.027515    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:07.048433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.041585    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:07.376275 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.376179    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:07.381009 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.380975    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:07.497382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.474373    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:07.497382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.474774    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:56:07.717433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.710860    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:56:07.742698 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.742661    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:56:07.837378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.801852    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:56:07.966415 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.852516    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:56:07.972897 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.972818    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:56:08.010379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.976462    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:56:08.011849 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.852931    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:56:08.012593 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.852957    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:56:08.016374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:08.013103    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:56:08.020371 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:08.016546    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:56:08.033373 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.856210    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:56:08.052658 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:08.052631    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:56:08.059568 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.856237    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:56:08.083395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.852908    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded"
Oct 30 12:56:08.126380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:07.852573    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:08.145666 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:08.083933    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:56:08.166968 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:08.166922    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:56:08.764577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:08.400720    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:08.764577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:08.400974    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:08.764577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:08.484015    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:56:08.764577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:08.658614    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:08.764577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:08.738493    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:56:08.959017 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:08.958970    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:08.963384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:08.959553    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:09.027449 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.015628    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:56:09.028564 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.028032    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:09.058497 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.058451    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:09.101387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.067921    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:56:09.110187 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.110154    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:56:09.118502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.118472    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:56:09.118723 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.118705    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:56:09.140470 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.140428    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:09.164381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.157238    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:09.620378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.546420    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:09.713387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.705899    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:09.744252 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.712629    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:56:09.761386 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.712668    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:56:10.005435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.761648    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:10.005435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.901424    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:56:10.005435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.984172    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:10.005435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:09.970438    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:56:10.040378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.039725    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:10.047545 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.040930    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:56:10.062482 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.058997    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:10.069424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.067936    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:10.163449 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.163399    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:10.222933 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.180903    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:10.487378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.476205    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:10.517413 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.480404    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:10.753249 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.753140    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:56:10.802575 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.802522    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:10.901381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.772707    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:56:10.901381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.895663    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:10.984378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.976989    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:56:11.155493 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:10.997905    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:56:11.224818 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:11.010404    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:56:11.225768 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:11.056630    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:11.244655 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:11.073529    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:56:11.248642 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:11.074287    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:11.273418 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:11.101650    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:56:11.273418 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:11.193558    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:11.273418 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:11.198801    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:11.273418 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:11.229562    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:11.640787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:11.640479    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:11.800538 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:11.800479    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:56:11.812069 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:11.812023    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:12.013437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.010488    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:56:12.013437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.010552    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:12.154536 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.154489    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:12.155335 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.155306    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:56:12.252379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.244383    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:12.256374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.254604    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:56:12.256374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.255919    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:12.256374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.256038    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:56:12.256374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.256347    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:56:12.256723 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.256611    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:12.262842 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.256837    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:56:12.262842 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.257675    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:12.625909 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.625862    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded"
Oct 30 12:56:12.807375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.801915    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:56:12.826656 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:12.826490    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:13.164730 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.163212    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:13.228474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.228400    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:13.276399 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.259474    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:13.276399 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.260988    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:13.276399 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.261271    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:56:13.276399 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.262846    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:13.276399 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.263128    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:56:13.277204 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.277170    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:13.277744 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.277718    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:56:13.425427 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.368121    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:13.553578 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.553539    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:13.688817 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.688726    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:56:13.823943 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.823893    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:56:13.833497 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:13.833441    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:14.242791 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.242744    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:14.243775 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.243741    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:14.275250 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.275208    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:56:14.275858 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.275826    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:56:14.277154 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.276507    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:56:14.278100 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.276538    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:56:14.278686 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.278660    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:56:14.310437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.297461    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:14.572730 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.572502    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:14.800687 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.800450    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:14.825499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.824876    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:14.836860 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.836429    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:14.877438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.876583    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:56:14.993545 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:14.991785    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:15.119747 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.119651    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:56:15.299374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.291497    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:15.344451 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.300171    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:56:15.345784 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.340745    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:15.346053 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.341697    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:56:15.346458 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.343516    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:15.347060 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.344134    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:56:15.347665 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.347645    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:15.349402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.349384    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:56:15.441518 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.423707    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:56:15.527586 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.523919    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:56:15.528276 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.528251    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:56:15.529546 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.529529    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:56:15.530098 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.530077    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:56:15.535103 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.531178    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:56:15.804067 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.804032    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:15.903527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:15.903491    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:16.005813 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.005759    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:16.420288 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.420255    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"]
Oct 30 12:56:16.424429 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.424406    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.441830 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.441798    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:16.444847 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.444825    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"]
Oct 30 12:56:16.452870 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.452848    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7a04e002-openshift-default-dockercfg-vxgwf\""
Oct 30 12:56:16.527928 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.527877    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-credential-socket\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.528347 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.528305    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0519800-4332-4a1c-94a8-205367079050-istio-podinfo\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.528609 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.528591    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv7v5\" (UniqueName: \"kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-kube-api-access-tv7v5\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.528773 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.528759    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-istio-token\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.528984 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.528969    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-certs\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.529171 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.529157    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-socket\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.529316 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.529302    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-data\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.529495 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.529481    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0519800-4332-4a1c-94a8-205367079050-istiod-ca-cert\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.529685 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.529669    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-envoy\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.632094 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.631885    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-istio-token\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.632667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.632479    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-certs\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.632667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.632602    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-socket\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.633892 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.632883    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-data\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.633892 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.632961    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0519800-4332-4a1c-94a8-205367079050-istiod-ca-cert\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.633892 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.633051    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-envoy\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.633892 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.633112    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-credential-socket\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.633892 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.633199    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0519800-4332-4a1c-94a8-205367079050-istio-podinfo\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.633892 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.633246    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv7v5\" (UniqueName: \"kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-kube-api-access-tv7v5\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.636517 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.634882    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-certs\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.636517 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.635491    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-socket\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.636517 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.635925    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-data\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.637127 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.637108    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0519800-4332-4a1c-94a8-205367079050-istiod-ca-cert\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.637929 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.637909    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-credential-socket\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.654709 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.654677    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0519800-4332-4a1c-94a8-205367079050-istio-podinfo\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.656913 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.656873    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-envoy\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.658065 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.658043    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-istio-token\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.658642 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.658620    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv7v5\" (UniqueName: \"kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-kube-api-access-tv7v5\") pod \"e2e-mt-7a04e002-openshift-default-6669855c89-9khb6\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") " pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:16.738531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:16.738465    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:17.095422 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:17.095398    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"]
Oct 30 12:56:17.098107 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:56:17.098076    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0519800_4332_4a1c_94a8_205367079050.slice/crio-cd6c26d8e039b9c169ecc5aa8ee53faed42ff9ca9828e4c4da93034ad0d03d50 WatchSource:0}: Error finding container cd6c26d8e039b9c169ecc5aa8ee53faed42ff9ca9828e4c4da93034ad0d03d50: Status 404 returned error can't find the container with id cd6c26d8e039b9c169ecc5aa8ee53faed42ff9ca9828e4c4da93034ad0d03d50
Oct 30 12:56:17.100790 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:17.100757    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:56:17.100879 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:17.100817    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:56:17.100879 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:17.100847    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:56:17.157882 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:17.157852    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6" event={"ID":"b0519800-4332-4a1c-94a8-205367079050","Type":"ContainerStarted","Data":"cd6c26d8e039b9c169ecc5aa8ee53faed42ff9ca9828e4c4da93034ad0d03d50"}
Oct 30 12:56:18.007858 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:18.007824    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:18.198834 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:18.198800    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6" event={"ID":"b0519800-4332-4a1c-94a8-205367079050","Type":"ContainerStarted","Data":"2b2a584aa0dba1b26666e4275189b70c39f088c4d05c2ac8e5e3e130ca4f915a"}
Oct 30 12:56:18.295735 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:18.295648    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6" podStartSLOduration=2.295624767 podStartE2EDuration="2.295624767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:18.255151235 +0000 UTC m=+1222.070477723" watchObservedRunningTime="2026-10-30 12:56:18.295624767 +0000 UTC m=+1222.110951259"
Oct 30 12:56:18.739625 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:18.739525    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:18.741218 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:18.741188    4522 patch_prober.go:29] interesting pod/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 12:56:18.741391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:18.741258    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6" podUID="b0519800-4332-4a1c-94a8-205367079050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 12:56:18.795546 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:18.795488    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:18.909043 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:18.909006    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:19.007817 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:19.007738    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:19.739100 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:19.739059    4522 patch_prober.go:29] interesting pod/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 12:56:19.739619 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:19.739133    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6" podUID="b0519800-4332-4a1c-94a8-205367079050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 12:56:20.048560 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:20.048482    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:20.708601 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:20.708557    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:20.738920 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:20.738869    4522 patch_prober.go:29] interesting pod/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 12:56:20.739213 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:20.739177    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6" podUID="b0519800-4332-4a1c-94a8-205367079050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 12:56:20.808613 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:20.808576    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:20.898655 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:20.898606    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"]
Oct 30 12:56:20.903457 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:20.903433    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"
Oct 30 12:56:20.925229 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:20.922810    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"]
Oct 30 12:56:20.925229 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:20.923327    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ef1186-dockercfg-2f2w7\""
Oct 30 12:56:20.925229 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:20.924208    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ef1186\""
Oct 30 12:56:20.970427 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:20.959564    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk8nc\" (UniqueName: \"kubernetes.io/projected/917fa195-0b4a-4cb8-8021-0573bcae153b-kube-api-access-zk8nc\") pod \"payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g\" (UID: \"917fa195-0b4a-4cb8-8021-0573bcae153b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"
Oct 30 12:56:20.970427 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:20.959857    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/917fa195-0b4a-4cb8-8021-0573bcae153b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g\" (UID: \"917fa195-0b4a-4cb8-8021-0573bcae153b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"
Oct 30 12:56:21.086377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.065527    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/917fa195-0b4a-4cb8-8021-0573bcae153b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g\" (UID: \"917fa195-0b4a-4cb8-8021-0573bcae153b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"
Oct 30 12:56:21.329905 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.329869    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk8nc\" (UniqueName: \"kubernetes.io/projected/917fa195-0b4a-4cb8-8021-0573bcae153b-kube-api-access-zk8nc\") pod \"payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g\" (UID: \"917fa195-0b4a-4cb8-8021-0573bcae153b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"
Oct 30 12:56:21.330872 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.325460    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"]
Oct 30 12:56:21.334519 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.070236    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/917fa195-0b4a-4cb8-8021-0573bcae153b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g\" (UID: \"917fa195-0b4a-4cb8-8021-0573bcae153b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"
Oct 30 12:56:21.345078 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.345056    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"
Oct 30 12:56:21.348033 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.348003    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"]
Oct 30 12:56:21.353965 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.352026    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk8nc\" (UniqueName: \"kubernetes.io/projected/917fa195-0b4a-4cb8-8021-0573bcae153b-kube-api-access-zk8nc\") pod \"payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g\" (UID: \"917fa195-0b4a-4cb8-8021-0573bcae153b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"
Oct 30 12:56:21.406814 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:56:21.406755    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:56:21.431928 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.431894    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53cab76d-b312-4afb-bc3d-c29ece805ce1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq\" (UID: \"53cab76d-b312-4afb-bc3d-c29ece805ce1\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"
Oct 30 12:56:21.432096 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.432040    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jksw\" (UniqueName: \"kubernetes.io/projected/53cab76d-b312-4afb-bc3d-c29ece805ce1-kube-api-access-5jksw\") pod \"payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq\" (UID: \"53cab76d-b312-4afb-bc3d-c29ece805ce1\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"
Oct 30 12:56:21.530319 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.530287    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"
Oct 30 12:56:21.533291 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.533099    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53cab76d-b312-4afb-bc3d-c29ece805ce1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq\" (UID: \"53cab76d-b312-4afb-bc3d-c29ece805ce1\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"
Oct 30 12:56:21.533291 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.533169    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jksw\" (UniqueName: \"kubernetes.io/projected/53cab76d-b312-4afb-bc3d-c29ece805ce1-kube-api-access-5jksw\") pod \"payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq\" (UID: \"53cab76d-b312-4afb-bc3d-c29ece805ce1\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"
Oct 30 12:56:21.534159 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.534132    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53cab76d-b312-4afb-bc3d-c29ece805ce1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq\" (UID: \"53cab76d-b312-4afb-bc3d-c29ece805ce1\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"
Oct 30 12:56:21.552636 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.552614    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jksw\" (UniqueName: \"kubernetes.io/projected/53cab76d-b312-4afb-bc3d-c29ece805ce1-kube-api-access-5jksw\") pod \"payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq\" (UID: \"53cab76d-b312-4afb-bc3d-c29ece805ce1\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"
Oct 30 12:56:21.692401 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.690596    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"
Oct 30 12:56:21.739587 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.739551    4522 patch_prober.go:29] interesting pod/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 12:56:21.740037 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.739608    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6" podUID="b0519800-4332-4a1c-94a8-205367079050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 12:56:21.858622 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:21.858322    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"]
Oct 30 12:56:21.858622 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:56:21.858580    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917fa195_0b4a_4cb8_8021_0573bcae153b.slice/crio-60fe0af1f7a12b3381469ea6d48a33748da0e2ce643e935bef6d6e926bb79382 WatchSource:0}: Error finding container 60fe0af1f7a12b3381469ea6d48a33748da0e2ce643e935bef6d6e926bb79382: Status 404 returned error can't find the container with id 60fe0af1f7a12b3381469ea6d48a33748da0e2ce643e935bef6d6e926bb79382
Oct 30 12:56:22.010374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.009995    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"]
Oct 30 12:56:22.010374 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:56:22.010033    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53cab76d_b312_4afb_bc3d_c29ece805ce1.slice/crio-f949b4cb0eeaf5571faf26bd7c3e102ae2d393a9fed764a1a24ee8c416fd9fa3 WatchSource:0}: Error finding container f949b4cb0eeaf5571faf26bd7c3e102ae2d393a9fed764a1a24ee8c416fd9fa3: Status 404 returned error can't find the container with id f949b4cb0eeaf5571faf26bd7c3e102ae2d393a9fed764a1a24ee8c416fd9fa3
Oct 30 12:56:22.149989 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.149923    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:22.361235 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.361203    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g" event={"ID":"917fa195-0b4a-4cb8-8021-0573bcae153b","Type":"ContainerStarted","Data":"d6a49bd9b1c0b2f98032c633d2b431c04e8adfca089d2c4219ba45e7f0ef0ac5"}
Oct 30 12:56:22.361419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.361249    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g" event={"ID":"917fa195-0b4a-4cb8-8021-0573bcae153b","Type":"ContainerStarted","Data":"60fe0af1f7a12b3381469ea6d48a33748da0e2ce643e935bef6d6e926bb79382"}
Oct 30 12:56:22.361878 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.361855    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"
Oct 30 12:56:22.363439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.363418    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq" event={"ID":"53cab76d-b312-4afb-bc3d-c29ece805ce1","Type":"ContainerStarted","Data":"0506fd950c009adb542f7ccb96adb95d9d155e214fc26d886d0f8b384f886c90"}
Oct 30 12:56:22.363529 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.363451    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq" event={"ID":"53cab76d-b312-4afb-bc3d-c29ece805ce1","Type":"ContainerStarted","Data":"f949b4cb0eeaf5571faf26bd7c3e102ae2d393a9fed764a1a24ee8c416fd9fa3"}
Oct 30 12:56:22.363850 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.363834    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"
Oct 30 12:56:22.388372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.376455    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:22.406879 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.406792    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g" podStartSLOduration=2.406778407 podStartE2EDuration="2.406778407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:22.40209126 +0000 UTC m=+1226.217417747" watchObservedRunningTime="2026-10-30 12:56:22.406778407 +0000 UTC m=+1226.222104935"
Oct 30 12:56:22.447267 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.447222    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq" podStartSLOduration=2.447209219 podStartE2EDuration="2.447209219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:22.444855534 +0000 UTC m=+1226.260182021" watchObservedRunningTime="2026-10-30 12:56:22.447209219 +0000 UTC m=+1226.262535704"
Oct 30 12:56:22.504380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.502896    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:22.743818 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.740716    4522 patch_prober.go:29] interesting pod/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 12:56:22.743818 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.740778    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6" podUID="b0519800-4332-4a1c-94a8-205367079050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 12:56:22.744682 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:22.744656    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:23.738810 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:23.738744    4522 patch_prober.go:29] interesting pod/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 12:56:23.738992 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:23.738803    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6" podUID="b0519800-4332-4a1c-94a8-205367079050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 12:56:24.085121 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:24.085084    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:24.406801 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:24.406721    4522 generic.go:356] "Generic (PLEG): container finished" podID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerID="14a2082ba02eabb2a0c76f0b2cbf661b84877f34cdd9d92eb18b7c5a030c8408" exitCode=0
Oct 30 12:56:24.406967 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:24.406805    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" event={"ID":"608e1d64-aa04-4bc1-a4a7-244a8c02317b","Type":"ContainerDied","Data":"14a2082ba02eabb2a0c76f0b2cbf661b84877f34cdd9d92eb18b7c5a030c8408"}
Oct 30 12:56:24.407514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:24.407493    4522 scope.go:122] "RemoveContainer" containerID="14a2082ba02eabb2a0c76f0b2cbf661b84877f34cdd9d92eb18b7c5a030c8408"
Oct 30 12:56:24.409007 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:24.408976    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:56:24.409106 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:24.409043    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:56:24.409106 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:24.409076    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:56:24.744161 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:24.744081    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:24.744583 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:24.744556    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:24.745267 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:24.745249    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 12:56:25.415190 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:25.415151    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" event={"ID":"608e1d64-aa04-4bc1-a4a7-244a8c02317b","Type":"ContainerStarted","Data":"77dc21cceb255c82efa34efa19ba65c414339deee14ca6ffae1c1b4db758c4a7"}
Oct 30 12:56:25.837811 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:25.837779    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:56:25.839219 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:25.839191    4522 patch_prober.go:29] interesting pod/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 12:56:25.839376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:25.839238    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 12:56:25.928497 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:25.928455    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:26.838160 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:26.838118    4522 patch_prober.go:29] interesting pod/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 12:56:26.838656 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:26.838180    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 12:56:27.784380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:27.781447    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:56:27.838391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:27.838343    4522 patch_prober.go:29] interesting pod/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 12:56:27.838779 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:27.838417    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 12:56:27.860921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:27.860893    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:28.838325 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:28.838294    4522 patch_prober.go:29] interesting pod/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 12:56:28.838525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:28.838385    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 12:56:29.708314 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:29.708277    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:29.846752 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:29.846716    4522 patch_prober.go:29] interesting pod/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 12:56:29.847364 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:29.847317    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 12:56:30.837798 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:30.837761    4522 patch_prober.go:29] interesting pod/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 12:56:30.837975 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:30.837822    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 12:56:31.511642 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:31.511607    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:31.842242 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:31.842212    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:56:31.843181 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:31.843159    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:56:32.040080 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:32.040044    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:33.301171 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:33.301136    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:34.295030 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:34.294971    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:34.393946 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:34.393905    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:34.411555 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:34.411525    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"
Oct 30 12:56:34.412865 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:34.412844    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"
Oct 30 12:56:35.445687 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:35.445656    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:37.012917 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.012871    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"]
Oct 30 12:56:37.017433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.017408    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.036704 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.036677    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"]
Oct 30 12:56:37.048370 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.045375    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ef1186-serving-cert\""
Oct 30 12:56:37.091960 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.091925    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:37.103471 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.103437    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-metrics-certs\") pod \"maas-api-e2e-api-a-ef1186-9654c9949-4rw4k\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.103663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.103532    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pflfw\" (UniqueName: \"kubernetes.io/projected/ba60f1fb-d268-4846-b604-3cd81bd289e3-kube-api-access-pflfw\") pod \"maas-api-e2e-api-a-ef1186-9654c9949-4rw4k\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.103663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.103613    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-maas-api-tls\") pod \"maas-api-e2e-api-a-ef1186-9654c9949-4rw4k\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.103774 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.103720    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba60f1fb-d268-4846-b604-3cd81bd289e3-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef1186-9654c9949-4rw4k\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.187400 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.187347    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:37.204885 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.204846    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pflfw\" (UniqueName: \"kubernetes.io/projected/ba60f1fb-d268-4846-b604-3cd81bd289e3-kube-api-access-pflfw\") pod \"maas-api-e2e-api-a-ef1186-9654c9949-4rw4k\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.205324 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.204940    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-maas-api-tls\") pod \"maas-api-e2e-api-a-ef1186-9654c9949-4rw4k\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.205324 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.205052    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba60f1fb-d268-4846-b604-3cd81bd289e3-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef1186-9654c9949-4rw4k\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.205324 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.205257    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-metrics-certs\") pod \"maas-api-e2e-api-a-ef1186-9654c9949-4rw4k\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.205777 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.205756    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba60f1fb-d268-4846-b604-3cd81bd289e3-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef1186-9654c9949-4rw4k\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.207500 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.207476    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-maas-api-tls\") pod \"maas-api-e2e-api-a-ef1186-9654c9949-4rw4k\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.207595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.207521    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-metrics-certs\") pod \"maas-api-e2e-api-a-ef1186-9654c9949-4rw4k\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.223607 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.223583    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pflfw\" (UniqueName: \"kubernetes.io/projected/ba60f1fb-d268-4846-b604-3cd81bd289e3-kube-api-access-pflfw\") pod \"maas-api-e2e-api-a-ef1186-9654c9949-4rw4k\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.305140 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.305100    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:37.337546 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.337488    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:37.740263 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:56:37.740230    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba60f1fb_d268_4846_b604_3cd81bd289e3.slice/crio-93986ab34c55ddeb0c6cc558ed808dc9d9a27bb8a50c0a4de9ca0480eb19c111 WatchSource:0}: Error finding container 93986ab34c55ddeb0c6cc558ed808dc9d9a27bb8a50c0a4de9ca0480eb19c111: Status 404 returned error can't find the container with id 93986ab34c55ddeb0c6cc558ed808dc9d9a27bb8a50c0a4de9ca0480eb19c111
Oct 30 12:56:37.740685 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.740664    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"]
Oct 30 12:56:37.961331 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:37.961289    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:38.565802 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:38.565747    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k" event={"ID":"ba60f1fb-d268-4846-b604-3cd81bd289e3","Type":"ContainerStarted","Data":"6bf591c9ac6a49ded37fa774270730eba82e28b6e86ef75fb8f3e10cb7ff43ee"}
Oct 30 12:56:38.566617 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:38.565829    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k" event={"ID":"ba60f1fb-d268-4846-b604-3cd81bd289e3","Type":"ContainerStarted","Data":"93986ab34c55ddeb0c6cc558ed808dc9d9a27bb8a50c0a4de9ca0480eb19c111"}
Oct 30 12:56:38.566617 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:38.566125    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:38.617454 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:38.611483    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k" podStartSLOduration=2.015022678 podStartE2EDuration="2.611461311s" totalImagesPullingTime="596.438633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:38.610505266 +0000 UTC m=+1242.425831753" watchObservedRunningTime="2026-10-30 12:56:38.611461311 +0000 UTC m=+1242.426787817"
Oct 30 12:56:38.921158 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:38.921061    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:40.522619 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:40.522590    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:40.813846 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:40.813762    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:42.499097 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:42.499059    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:44.065165 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:44.065128    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"]
Oct 30 12:56:44.099167 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:44.099139    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"]
Oct 30 12:56:44.194924 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:44.194879    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:44.295200 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:44.295172    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:44.397017 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:44.396933    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:44.578382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:44.578340    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:56:45.978055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:45.978019    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:47.659158 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:47.659118    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:48.320921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:48.320890    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:49.066252 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.066217    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn" podUID="59cc37e9-f732-42ef-89c6-638f7c1948f9" containerName="payload-processing" containerID="cri-o://41d4c9664aff662b5e4d18ced0d61dd195883a576c507dfc4c59ffba63f427f6" gracePeriod=25
Oct 30 12:56:49.099965 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.099930    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx" podUID="37969ed0-570c-4d9f-b726-5dbdb2596560" containerName="payload-pre-processing" containerID="cri-o://baebc623196d985279533c16316ddcfb9689f6f930c092223b85b500e5278833" gracePeriod=25
Oct 30 12:56:49.341918 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.341834    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:49.398638 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.398609    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"
Oct 30 12:56:49.474776 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.467961    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59cc37e9-f732-42ef-89c6-638f7c1948f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59cc37e9-f732-42ef-89c6-638f7c1948f9-plugins-config-volume\") pod \"59cc37e9-f732-42ef-89c6-638f7c1948f9\" (UID: \"59cc37e9-f732-42ef-89c6-638f7c1948f9\") "
Oct 30 12:56:49.474776 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.468272    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59cc37e9-f732-42ef-89c6-638f7c1948f9-kube-api-access-djvnh\" (UniqueName: \"kubernetes.io/projected/59cc37e9-f732-42ef-89c6-638f7c1948f9-kube-api-access-djvnh\") pod \"59cc37e9-f732-42ef-89c6-638f7c1948f9\" (UID: \"59cc37e9-f732-42ef-89c6-638f7c1948f9\") "
Oct 30 12:56:49.474776 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.474699    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59cc37e9-f732-42ef-89c6-638f7c1948f9-plugins-config-volume" pod "59cc37e9-f732-42ef-89c6-638f7c1948f9" (UID: "59cc37e9-f732-42ef-89c6-638f7c1948f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:56:49.495805 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.494543    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cc37e9-f732-42ef-89c6-638f7c1948f9-kube-api-access-djvnh" pod "59cc37e9-f732-42ef-89c6-638f7c1948f9" (UID: "59cc37e9-f732-42ef-89c6-638f7c1948f9"). InnerVolumeSpecName "kube-api-access-djvnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:56:49.564247 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.564095    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"
Oct 30 12:56:49.569920 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.569897    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djvnh\" (UniqueName: \"kubernetes.io/projected/59cc37e9-f732-42ef-89c6-638f7c1948f9-kube-api-access-djvnh\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:56:49.570090 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.570070    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59cc37e9-f732-42ef-89c6-638f7c1948f9-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:56:49.671254 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.671220    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37969ed0-570c-4d9f-b726-5dbdb2596560-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37969ed0-570c-4d9f-b726-5dbdb2596560-plugins-config-volume\") pod \"37969ed0-570c-4d9f-b726-5dbdb2596560\" (UID: \"37969ed0-570c-4d9f-b726-5dbdb2596560\") "
Oct 30 12:56:49.671484 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.671469    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37969ed0-570c-4d9f-b726-5dbdb2596560-kube-api-access-57r9l\" (UniqueName: \"kubernetes.io/projected/37969ed0-570c-4d9f-b726-5dbdb2596560-kube-api-access-57r9l\") pod \"37969ed0-570c-4d9f-b726-5dbdb2596560\" (UID: \"37969ed0-570c-4d9f-b726-5dbdb2596560\") "
Oct 30 12:56:49.671668 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.671641    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37969ed0-570c-4d9f-b726-5dbdb2596560-plugins-config-volume" pod "37969ed0-570c-4d9f-b726-5dbdb2596560" (UID: "37969ed0-570c-4d9f-b726-5dbdb2596560"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:56:49.671945 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.671926    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37969ed0-570c-4d9f-b726-5dbdb2596560-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:56:49.673590 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.673564    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37969ed0-570c-4d9f-b726-5dbdb2596560-kube-api-access-57r9l" pod "37969ed0-570c-4d9f-b726-5dbdb2596560" (UID: "37969ed0-570c-4d9f-b726-5dbdb2596560"). InnerVolumeSpecName "kube-api-access-57r9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:56:49.678411 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.678388    4522 generic.go:356] "Generic (PLEG): container finished" podID="59cc37e9-f732-42ef-89c6-638f7c1948f9" containerID="41d4c9664aff662b5e4d18ced0d61dd195883a576c507dfc4c59ffba63f427f6" exitCode=0
Oct 30 12:56:49.678514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.678479    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"
Oct 30 12:56:49.678577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.678512    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn" event={"ID":"59cc37e9-f732-42ef-89c6-638f7c1948f9","Type":"ContainerDied","Data":"41d4c9664aff662b5e4d18ced0d61dd195883a576c507dfc4c59ffba63f427f6"}
Oct 30 12:56:49.678577 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.678564    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn" event={"ID":"59cc37e9-f732-42ef-89c6-638f7c1948f9","Type":"ContainerDied","Data":"4f88b3d8238dd12502c67239ccd8cee1af5976b41f5c1c45ded55c4977d15c7e"}
Oct 30 12:56:49.678663 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.678592    4522 scope.go:122] "RemoveContainer" containerID="41d4c9664aff662b5e4d18ced0d61dd195883a576c507dfc4c59ffba63f427f6"
Oct 30 12:56:49.679972 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.679953    4522 generic.go:356] "Generic (PLEG): container finished" podID="37969ed0-570c-4d9f-b726-5dbdb2596560" containerID="baebc623196d985279533c16316ddcfb9689f6f930c092223b85b500e5278833" exitCode=0
Oct 30 12:56:49.680068 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.679981    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx" event={"ID":"37969ed0-570c-4d9f-b726-5dbdb2596560","Type":"ContainerDied","Data":"baebc623196d985279533c16316ddcfb9689f6f930c092223b85b500e5278833"}
Oct 30 12:56:49.680068 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.680007    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"
Oct 30 12:56:49.680068 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.680043    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx" event={"ID":"37969ed0-570c-4d9f-b726-5dbdb2596560","Type":"ContainerDied","Data":"209434b29ace7345afae7604abb6dd0b7323f6e874d1d6885b9fbf17f1a1e6fb"}
Oct 30 12:56:49.689589 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.689566    4522 scope.go:122] "RemoveContainer" containerID="41d4c9664aff662b5e4d18ced0d61dd195883a576c507dfc4c59ffba63f427f6"
Oct 30 12:56:49.689896 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:56:49.689874    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d4c9664aff662b5e4d18ced0d61dd195883a576c507dfc4c59ffba63f427f6\": container with ID starting with 41d4c9664aff662b5e4d18ced0d61dd195883a576c507dfc4c59ffba63f427f6 not found: ID does not exist" containerID="41d4c9664aff662b5e4d18ced0d61dd195883a576c507dfc4c59ffba63f427f6"
Oct 30 12:56:49.689984 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.689907    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d4c9664aff662b5e4d18ced0d61dd195883a576c507dfc4c59ffba63f427f6"} err="failed to get container status \"41d4c9664aff662b5e4d18ced0d61dd195883a576c507dfc4c59ffba63f427f6\": rpc error: code = NotFound desc = could not find container \"41d4c9664aff662b5e4d18ced0d61dd195883a576c507dfc4c59ffba63f427f6\": container with ID starting with 41d4c9664aff662b5e4d18ced0d61dd195883a576c507dfc4c59ffba63f427f6 not found: ID does not exist"
Oct 30 12:56:49.689984 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.689930    4522 scope.go:122] "RemoveContainer" containerID="baebc623196d985279533c16316ddcfb9689f6f930c092223b85b500e5278833"
Oct 30 12:56:49.698540 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.698519    4522 scope.go:122] "RemoveContainer" containerID="baebc623196d985279533c16316ddcfb9689f6f930c092223b85b500e5278833"
Oct 30 12:56:49.698787 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:56:49.698769    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baebc623196d985279533c16316ddcfb9689f6f930c092223b85b500e5278833\": container with ID starting with baebc623196d985279533c16316ddcfb9689f6f930c092223b85b500e5278833 not found: ID does not exist" containerID="baebc623196d985279533c16316ddcfb9689f6f930c092223b85b500e5278833"
Oct 30 12:56:49.698838 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.698794    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baebc623196d985279533c16316ddcfb9689f6f930c092223b85b500e5278833"} err="failed to get container status \"baebc623196d985279533c16316ddcfb9689f6f930c092223b85b500e5278833\": rpc error: code = NotFound desc = could not find container \"baebc623196d985279533c16316ddcfb9689f6f930c092223b85b500e5278833\": container with ID starting with baebc623196d985279533c16316ddcfb9689f6f930c092223b85b500e5278833 not found: ID does not exist"
Oct 30 12:56:49.729217 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.729193    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"]
Oct 30 12:56:49.732601 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.732583    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6bda55-5d47b6f544-wn6rn"]
Oct 30 12:56:49.766633 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.766609    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"]
Oct 30 12:56:49.769810 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.769785    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6bda55-6d56d87b79-zgwdx"]
Oct 30 12:56:49.773240 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:49.773220    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57r9l\" (UniqueName: \"kubernetes.io/projected/37969ed0-570c-4d9f-b726-5dbdb2596560-kube-api-access-57r9l\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:56:50.334725 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:50.334652    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37969ed0-570c-4d9f-b726-5dbdb2596560" path="/var/lib/kubelet/pods/37969ed0-570c-4d9f-b726-5dbdb2596560/volumes"
Oct 30 12:56:50.335137 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:50.335120    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59cc37e9-f732-42ef-89c6-638f7c1948f9" path="/var/lib/kubelet/pods/59cc37e9-f732-42ef-89c6-638f7c1948f9/volumes"
Oct 30 12:56:51.089299 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:51.089262    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:51.134313 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:56:51.134279    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:56:52.716409 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:52.716375    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:54.464647 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:54.464609    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:56.407072 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:56.407043    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:58.121467 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:58.121418    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:59.354134 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:59.354103    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:59.722385 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:56:59.722295    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:00.028618 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:00.028582    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:00.515094 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:00.515013    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:00.617536 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:00.617497    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:01.283075 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:01.283039    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:01.959075 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:01.959038    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:02.261432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:02.261306    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:02.356302 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:02.356191    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:02.638608 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:02.638573    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:04.062079 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:04.062043    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:04.495694 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:04.495610    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:04.603885 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:04.603837    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:05.816848 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:05.816813    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:06.176221 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:06.176143    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:06.277217 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:06.277176    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:07.376998 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:07.376967    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:08.084300 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:08.084264    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:08.191683 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:08.191650    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:08.274491 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:08.274460    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:08.699192 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:08.699161    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:08.802615 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:08.802578    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:09.078745 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:09.078704    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"]
Oct 30 12:57:09.079055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:09.079022    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerName="istio-proxy" containerID="cri-o://77dc21cceb255c82efa34efa19ba65c414339deee14ca6ffae1c1b4db758c4a7" gracePeriod=30
Oct 30 12:57:09.083942 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:09.083900    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:10.312548 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:10.312508    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:11.975738 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:11.975704    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:57:12.002990 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:12.002949    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:12.603374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:12.601379    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:13.122326 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:13.122287    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:13.279376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:13.273903    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:14.453592 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.453469    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:14.639314 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.608891    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.805269    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-credential-socket\") pod \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") "
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.805342    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-socket\") pod \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") "
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.807060    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-kube-api-access-kf92n\" (UniqueName: \"kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-kube-api-access-kf92n\") pod \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") "
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.807183    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istiod-ca-cert\") pod \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") "
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.807291    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-token\" (UniqueName: \"kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-token\") pod \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") "
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.807374    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-data\") pod \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") "
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.807440    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-podinfo\") pod \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") "
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.807498    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-envoy\") pod \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") "
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.807573    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-certs\") pod \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\" (UID: \"608e1d64-aa04-4bc1-a4a7-244a8c02317b\") "
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.823399    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-certs" pod "608e1d64-aa04-4bc1-a4a7-244a8c02317b" (UID: "608e1d64-aa04-4bc1-a4a7-244a8c02317b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.844929    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-data" pod "608e1d64-aa04-4bc1-a4a7-244a8c02317b" (UID: "608e1d64-aa04-4bc1-a4a7-244a8c02317b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.845398    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istiod-ca-cert" pod "608e1d64-aa04-4bc1-a4a7-244a8c02317b" (UID: "608e1d64-aa04-4bc1-a4a7-244a8c02317b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.855897    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.855925    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.855940    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.864532    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-socket" pod "608e1d64-aa04-4bc1-a4a7-244a8c02317b" (UID: "608e1d64-aa04-4bc1-a4a7-244a8c02317b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.868434    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-credential-socket" pod "608e1d64-aa04-4bc1-a4a7-244a8c02317b" (UID: "608e1d64-aa04-4bc1-a4a7-244a8c02317b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:14.892760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:14.880738    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-token" pod "608e1d64-aa04-4bc1-a4a7-244a8c02317b" (UID: "608e1d64-aa04-4bc1-a4a7-244a8c02317b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:15.008082 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.008042    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:15.009127 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.009099    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:15.009375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.009334    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:15.028758 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.027963    4522 generic.go:356] "Generic (PLEG): container finished" podID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerID="77dc21cceb255c82efa34efa19ba65c414339deee14ca6ffae1c1b4db758c4a7" exitCode=0
Oct 30 12:57:15.028758 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.028079    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" event={"ID":"608e1d64-aa04-4bc1-a4a7-244a8c02317b","Type":"ContainerDied","Data":"77dc21cceb255c82efa34efa19ba65c414339deee14ca6ffae1c1b4db758c4a7"}
Oct 30 12:57:15.028758 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.028133    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d" event={"ID":"608e1d64-aa04-4bc1-a4a7-244a8c02317b","Type":"ContainerDied","Data":"bcd148145aa79bfa03bf2c476a52b016adba3e7ce776dae7208cc43969081075"}
Oct 30 12:57:15.028758 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.028167    4522 scope.go:122] "RemoveContainer" containerID="77dc21cceb255c82efa34efa19ba65c414339deee14ca6ffae1c1b4db758c4a7"
Oct 30 12:57:15.028758 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.028339    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"
Oct 30 12:57:15.038703 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.038602    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-kube-api-access-kf92n" pod "608e1d64-aa04-4bc1-a4a7-244a8c02317b" (UID: "608e1d64-aa04-4bc1-a4a7-244a8c02317b"). InnerVolumeSpecName "kube-api-access-kf92n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:15.123962 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.123927    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf92n\" (UniqueName: \"kubernetes.io/projected/608e1d64-aa04-4bc1-a4a7-244a8c02317b-kube-api-access-kf92n\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:15.149045 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.148731    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-podinfo" pod "608e1d64-aa04-4bc1-a4a7-244a8c02317b" (UID: "608e1d64-aa04-4bc1-a4a7-244a8c02317b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:57:15.174976 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.174929    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-envoy" pod "608e1d64-aa04-4bc1-a4a7-244a8c02317b" (UID: "608e1d64-aa04-4bc1-a4a7-244a8c02317b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:15.175317 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.175296    4522 scope.go:122] "RemoveContainer" containerID="14a2082ba02eabb2a0c76f0b2cbf661b84877f34cdd9d92eb18b7c5a030c8408"
Oct 30 12:57:15.274831 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.250601    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:15.274831 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.250633    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e1d64-aa04-4bc1-a4a7-244a8c02317b-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:15.274831 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.256058    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:15.312439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.312011    4522 scope.go:122] "RemoveContainer" containerID="77dc21cceb255c82efa34efa19ba65c414339deee14ca6ffae1c1b4db758c4a7"
Oct 30 12:57:15.313133 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:57:15.312760    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77dc21cceb255c82efa34efa19ba65c414339deee14ca6ffae1c1b4db758c4a7\": container with ID starting with 77dc21cceb255c82efa34efa19ba65c414339deee14ca6ffae1c1b4db758c4a7 not found: ID does not exist" containerID="77dc21cceb255c82efa34efa19ba65c414339deee14ca6ffae1c1b4db758c4a7"
Oct 30 12:57:15.313133 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.312800    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77dc21cceb255c82efa34efa19ba65c414339deee14ca6ffae1c1b4db758c4a7"} err="failed to get container status \"77dc21cceb255c82efa34efa19ba65c414339deee14ca6ffae1c1b4db758c4a7\": rpc error: code = NotFound desc = could not find container \"77dc21cceb255c82efa34efa19ba65c414339deee14ca6ffae1c1b4db758c4a7\": container with ID starting with 77dc21cceb255c82efa34efa19ba65c414339deee14ca6ffae1c1b4db758c4a7 not found: ID does not exist"
Oct 30 12:57:15.313133 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.312829    4522 scope.go:122] "RemoveContainer" containerID="14a2082ba02eabb2a0c76f0b2cbf661b84877f34cdd9d92eb18b7c5a030c8408"
Oct 30 12:57:15.313563 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:57:15.313469    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a2082ba02eabb2a0c76f0b2cbf661b84877f34cdd9d92eb18b7c5a030c8408\": container with ID starting with 14a2082ba02eabb2a0c76f0b2cbf661b84877f34cdd9d92eb18b7c5a030c8408 not found: ID does not exist" containerID="14a2082ba02eabb2a0c76f0b2cbf661b84877f34cdd9d92eb18b7c5a030c8408"
Oct 30 12:57:15.313563 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.313512    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a2082ba02eabb2a0c76f0b2cbf661b84877f34cdd9d92eb18b7c5a030c8408"} err="failed to get container status \"14a2082ba02eabb2a0c76f0b2cbf661b84877f34cdd9d92eb18b7c5a030c8408\": rpc error: code = NotFound desc = could not find container \"14a2082ba02eabb2a0c76f0b2cbf661b84877f34cdd9d92eb18b7c5a030c8408\": container with ID starting with 14a2082ba02eabb2a0c76f0b2cbf661b84877f34cdd9d92eb18b7c5a030c8408 not found: ID does not exist"
Oct 30 12:57:15.360445 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.360402    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:15.377959 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.377821    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:15.600536 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.600499    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"]
Oct 30 12:57:15.640924 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.640882    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:15.691430 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.685463    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f9e9fd-openshift-default-6cccc5769c-wwz9d"]
Oct 30 12:57:15.786854 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.774390    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:15.786854 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.775005    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:15.786854 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.775062    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:57:15.849908 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:15.848456    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.059382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.057999    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.059382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.059103    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.088089 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.061272    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.178397 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.170535    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.220527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.207173    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:16.220527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.207291    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2" podUID="c3a4b170-e0c7-4b26-8817-aa969962c106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.240724 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.240674    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.255379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.251959    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.292378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.286446    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.379439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.379069    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:57:16.560268 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.560223    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" path="/var/lib/kubelet/pods/608e1d64-aa04-4bc1-a4a7-244a8c02317b/volumes"
Oct 30 12:57:16.561432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.561411    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:57:16.780441 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.779631    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.785722 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.785682    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:57:16.853494 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.853450    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.865440 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.854243    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:57:16.865440 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.856603    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:16.865440 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:16.857067    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:57:17.076555 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.076454    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:17.082193 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.077101    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:17.104699 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.103703    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:57:17.104858 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.104739    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:57:17.105499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.105467    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:57:17.105911 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.105886    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:57:17.163103 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.162588    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:17.163103 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.162646    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:17.186391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.185204    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:57:17.186391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.185655    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:57:17.255587 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.255556    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:57:17.316375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.282580    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:57:17.316375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.283085    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:57:17.400444 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.383922    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:17.400831 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.400475    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:17.401267 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.400516    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:57:17.423774 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.400537    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:57:17.437851 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.437817    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:57:17.438113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.438096    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:57:17.531920 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.531890    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:57:17.574394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.574341    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:17.574765 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.574732    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:17.690710 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.690619    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:17.847505 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.780384    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:17.867733 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:17.867695    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:57:18.123982 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:18.123935    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:57:18.173416 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:18.165892    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:18.339406 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:18.194235    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:18.351377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:18.343439    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:18.359376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:18.265476    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:18.411435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:18.194306    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:18.424381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:18.304678    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:57:18.442769 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:18.411704    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:18.492927 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:18.411742    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:57:18.507697 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:18.446291    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:19.109007 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:18.763511    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:57:19.140318 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:18.866224    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:19.210913 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:18.900536    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:19.236775 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:19.167309    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:19.338985 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:19.338936    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:19.417674 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:19.376871    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:19.421283 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:19.421230    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:19.422766 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:19.422743    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:57:19.502376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:19.492925    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:19.508963 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:19.508901    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:19.671451 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:19.579303    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded"
Oct 30 12:57:19.797017 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:19.796976    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:20.107937 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.107878    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:20.151565 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.151518    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:57:20.152670 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.152646    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:57:20.219519 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.219469    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:57:20.240225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.240178    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:20.260544 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.260500    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:57:20.465271 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.465155    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:20.466478 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.466425    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:57:20.467204 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.467183    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:20.483375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.467384    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:20.488506 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.486426    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:20.527570 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.512458    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:20.668379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.535737    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:57:20.670040 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.651674    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:20.804395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:20.799746    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.035421 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.015100    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:21.040566 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.018454    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:21.049399 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.045424    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.116437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.109527    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:57:21.172391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.153533    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.209661 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.207868    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.241529 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.241089    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.244291 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.242470    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.263150 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.263068    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.521104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.516802    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.521104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.517392    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.521104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.517554    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.647410 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.647077    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.672323 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.671454    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.697792 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.696857    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:21.804562 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.804445    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:57:21.872762 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:57:21.872720    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:57:22.007378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.995291    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:22.007378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:21.995377    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:22.110560 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.110452    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:22.162325 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.162282    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:22.256503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.256451    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:57:22.257451 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.257415    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:22.275581 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.275135    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:57:22.390107 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.389998    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:22.500606 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.500539    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:22.518580 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.518521    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:57:22.519277 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.519237    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:22.530460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.530407    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded"
Oct 30 12:57:22.675434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.674956    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:57:22.689379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.675983    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:22.697478 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.697432    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:57:22.820384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:22.808455    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:23.114514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:23.114462    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:23.175708 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:23.175610    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:57:23.206714 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:23.206683    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:57:23.295378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:23.295313    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:23.466367 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:23.466072    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:57:23.705514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:23.698030    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:23.705514 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:23.524567    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:23.806176 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:23.723421    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:57:23.806176 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:23.524865    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:23.806176 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:23.724856    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:23.815694 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:23.811474    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:24.122872 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.121596    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:24.220379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.177662    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:24.341415 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.336477    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:24.459477 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.430966    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 30 12:57:24.700505 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.700455    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:24.704980 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.704945    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:57:24.717653 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.717572    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:57:24.732484 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.732444    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:24.732918 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.732889    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:57:24.756528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.733522    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:24.776655 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.758133    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:57:24.821968 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.821925    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:24.871555 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.822228    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded"
Oct 30 12:57:24.931536 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:24.929730    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:25.044509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.027745    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Oct 30 12:57:25.044509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.028454    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:25.118454 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.118413    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"]
Oct 30 12:57:25.127390 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.119665    4522 cpu_manager.go:414] "containerMap: removing container" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerName="istio-proxy"
Oct 30 12:57:25.152687 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.152656    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerName="istio-proxy"
Oct 30 12:57:25.171465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.170030    4522 cpu_manager.go:414] "containerMap: removing container" podUID="59cc37e9-f732-42ef-89c6-638f7c1948f9" containerName="payload-processing"
Oct 30 12:57:25.171465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.170076    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59cc37e9-f732-42ef-89c6-638f7c1948f9" containerName="payload-processing"
Oct 30 12:57:25.171465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.170149    4522 cpu_manager.go:414] "containerMap: removing container" podUID="37969ed0-570c-4d9f-b726-5dbdb2596560" containerName="payload-pre-processing"
Oct 30 12:57:25.171465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.170157    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37969ed0-570c-4d9f-b726-5dbdb2596560" containerName="payload-pre-processing"
Oct 30 12:57:25.171465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.170428    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerName="istio-proxy"
Oct 30 12:57:25.171465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.170449    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="59cc37e9-f732-42ef-89c6-638f7c1948f9" containerName="payload-processing"
Oct 30 12:57:25.171465 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.170469    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="37969ed0-570c-4d9f-b726-5dbdb2596560" containerName="payload-pre-processing"
Oct 30 12:57:25.199424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.127664    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:57:25.238458 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.238417    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:57:25.416212 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.416127    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"
Oct 30 12:57:25.468987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.463484    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2bfb1b\""
Oct 30 12:57:25.468987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.463611    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2bfb1b-dockercfg-k7lm2\""
Oct 30 12:57:25.468987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.464868    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:25.468987 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.466437    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:57:25.599384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.599149    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqxj6\" (UniqueName: \"kubernetes.io/projected/5fb623f7-f001-4d88-b898-4e50b421b05b-kube-api-access-zqxj6\") pod \"payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn\" (UID: \"5fb623f7-f001-4d88-b898-4e50b421b05b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"
Oct 30 12:57:25.599585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.599463    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb623f7-f001-4d88-b898-4e50b421b05b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn\" (UID: \"5fb623f7-f001-4d88-b898-4e50b421b05b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"
Oct 30 12:57:25.723716 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.723616    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"]
Oct 30 12:57:25.752107 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.752066    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:57:25.754851 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.754815    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:57:25.767188 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.767141    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:57:25.777378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.768681    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb623f7-f001-4d88-b898-4e50b421b05b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn\" (UID: \"5fb623f7-f001-4d88-b898-4e50b421b05b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"
Oct 30 12:57:25.787610 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.787567    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqxj6\" (UniqueName: \"kubernetes.io/projected/5fb623f7-f001-4d88-b898-4e50b421b05b-kube-api-access-zqxj6\") pod \"payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn\" (UID: \"5fb623f7-f001-4d88-b898-4e50b421b05b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"
Oct 30 12:57:25.789896 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.789865    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb623f7-f001-4d88-b898-4e50b421b05b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn\" (UID: \"5fb623f7-f001-4d88-b898-4e50b421b05b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"
Oct 30 12:57:25.911745 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.822753    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:57:25.916618 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.916575    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7145-86b048"]
Oct 30 12:57:25.918868 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.918832    4522 cpu_manager.go:414] "containerMap: removing container" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerName="istio-proxy"
Oct 30 12:57:25.919036 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.919018    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerName="istio-proxy"
Oct 30 12:57:25.920032 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.920008    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="608e1d64-aa04-4bc1-a4a7-244a8c02317b" containerName="istio-proxy"
Oct 30 12:57:25.925597 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.925564    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:25.929261 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.927158    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded"
Oct 30 12:57:25.929434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.927191    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:57:25.930557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.927217    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7145-86b048"
Oct 30 12:57:25.959603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.936785    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqxj6\" (UniqueName: \"kubernetes.io/projected/5fb623f7-f001-4d88-b898-4e50b421b05b-kube-api-access-zqxj6\") pod \"payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn\" (UID: \"5fb623f7-f001-4d88-b898-4e50b421b05b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"
Oct 30 12:57:25.959603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.939688    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"
Oct 30 12:57:25.959603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.943699    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:57:25.959603 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.944042    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:57:26.010470 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:25.982243    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7kfp9\""
Oct 30 12:57:26.088955 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.088903    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:26.173126 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.110684    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:26.173126 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.111024    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:57:26.173126 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.116234    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvw7l\" (UniqueName: \"kubernetes.io/projected/061a8bbd-ce4a-48db-9024-f402fcc6994d-kube-api-access-hvw7l\") pod \"test-curl-7145-86b048\" (UID: \"061a8bbd-ce4a-48db-9024-f402fcc6994d\") " pod="opendatahub/test-curl-7145-86b048"
Oct 30 12:57:26.173126 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.143444    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:57:26.236144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.194982    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:57:26.236144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.206479    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:57:26.236144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.213079    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7145-86b048"]
Oct 30 12:57:26.236144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.223391    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvw7l\" (UniqueName: \"kubernetes.io/projected/061a8bbd-ce4a-48db-9024-f402fcc6994d-kube-api-access-hvw7l\") pod \"test-curl-7145-86b048\" (UID: \"061a8bbd-ce4a-48db-9024-f402fcc6994d\") " pod="opendatahub/test-curl-7145-86b048"
Oct 30 12:57:26.243389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.243329    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:57:26.252541 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.252516    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvw7l\" (UniqueName: \"kubernetes.io/projected/061a8bbd-ce4a-48db-9024-f402fcc6994d-kube-api-access-hvw7l\") pod \"test-curl-7145-86b048\" (UID: \"061a8bbd-ce4a-48db-9024-f402fcc6994d\") " pod="opendatahub/test-curl-7145-86b048"
Oct 30 12:57:26.355820 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.355790    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:57:26.356180 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.356162    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:57:26.356368 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.356337    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:57:26.356515 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.356501    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:57:26.356641 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.356630    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:57:26.356752 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.356741    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:57:26.358364 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.356917    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:57:26.358364 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.356955    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:57:26.493612 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.493545    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7145-86b048"
Oct 30 12:57:26.520831 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.520804    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"]
Oct 30 12:57:26.521452 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:57:26.521423    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb623f7_f001_4d88_b898_4e50b421b05b.slice/crio-ca3fa8dc343c561abcb035d04f5e5f07bf243cffdfd18920da344e5c4825146a WatchSource:0}: Error finding container ca3fa8dc343c561abcb035d04f5e5f07bf243cffdfd18920da344e5c4825146a: Status 404 returned error can't find the container with id ca3fa8dc343c561abcb035d04f5e5f07bf243cffdfd18920da344e5c4825146a
Oct 30 12:57:26.823218 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.823178    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:26.837989 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:57:26.837961    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod061a8bbd_ce4a_48db_9024_f402fcc6994d.slice/crio-856ae8ec01a3687cbcb6efb5b90d9528238faa42df6f71f730a05979c35b07d6 WatchSource:0}: Error finding container 856ae8ec01a3687cbcb6efb5b90d9528238faa42df6f71f730a05979c35b07d6: Status 404 returned error can't find the container with id 856ae8ec01a3687cbcb6efb5b90d9528238faa42df6f71f730a05979c35b07d6
Oct 30 12:57:26.839481 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:26.839457    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7145-86b048"]
Oct 30 12:57:27.092935 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:27.092896    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7145-86b048" event={"ID":"061a8bbd-ce4a-48db-9024-f402fcc6994d","Type":"ContainerStarted","Data":"856ae8ec01a3687cbcb6efb5b90d9528238faa42df6f71f730a05979c35b07d6"}
Oct 30 12:57:27.094993 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:27.094953    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn" event={"ID":"5fb623f7-f001-4d88-b898-4e50b421b05b","Type":"ContainerStarted","Data":"6416ca5e28d5c38cd725e39b0c0fcd60d6b25ea3b0f2266ae3cb7b3026b57b6d"}
Oct 30 12:57:27.095124 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:27.095010    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn" event={"ID":"5fb623f7-f001-4d88-b898-4e50b421b05b","Type":"ContainerStarted","Data":"ca3fa8dc343c561abcb035d04f5e5f07bf243cffdfd18920da344e5c4825146a"}
Oct 30 12:57:27.095255 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:27.095240    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"
Oct 30 12:57:27.134143 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:27.134089    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn" podStartSLOduration=2.134073485 podStartE2EDuration="2.134073485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:27.132125799 +0000 UTC m=+1290.947452288" watchObservedRunningTime="2026-10-30 12:57:27.134073485 +0000 UTC m=+1290.949399999"
Oct 30 12:57:28.504611 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.504536    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"]
Oct 30 12:57:28.514069 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.514011    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:28.532429 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.532398    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-70621c-serving-cert\""
Oct 30 12:57:28.553245 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.553211    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"]
Oct 30 12:57:28.584858 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.584814    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:28.603601 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.603571    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9n6j\" (UniqueName: \"kubernetes.io/projected/e7580498-3df4-4052-b222-4020e493c364-kube-api-access-n9n6j\") pod \"maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:28.603759 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.603633    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7580498-3df4-4052-b222-4020e493c364-openshift-service-ca\") pod \"maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:28.603759 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.603753    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-metrics-certs\") pod \"maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:28.603893 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.603867    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-maas-api-tls\") pod \"maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:28.710873 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.706415    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-metrics-certs\") pod \"maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:28.710873 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.706582    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-maas-api-tls\") pod \"maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:28.710873 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.706703    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9n6j\" (UniqueName: \"kubernetes.io/projected/e7580498-3df4-4052-b222-4020e493c364-kube-api-access-n9n6j\") pod \"maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:28.710873 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.706763    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7580498-3df4-4052-b222-4020e493c364-openshift-service-ca\") pod \"maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:28.710873 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.707738    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7580498-3df4-4052-b222-4020e493c364-openshift-service-ca\") pod \"maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:28.717373 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:57:28.713506    4522 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-serving-cert: secret "maas-api-e2e-api-b-70621c-serving-cert" not found
Oct 30 12:57:28.717373 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:57:28.713595    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-maas-api-tls podName:e7580498-3df4-4052-b222-4020e493c364 nodeName:}" failed. No retries permitted until 2026-10-30 12:57:29.213569313 +0000 UTC m=+1293.028895783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-maas-api-tls") pod "maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv" (UID: "e7580498-3df4-4052-b222-4020e493c364") : secret "maas-api-e2e-api-b-70621c-serving-cert" not found
Oct 30 12:57:28.752432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.736857    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:28.752432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.740771    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9n6j\" (UniqueName: \"kubernetes.io/projected/e7580498-3df4-4052-b222-4020e493c364-kube-api-access-n9n6j\") pod \"maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:28.781372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.778383    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-metrics-certs\") pod \"maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:28.927907 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:28.927865    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:29.215574 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:29.215494    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-maas-api-tls\") pod \"maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:29.217975 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:29.217944    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-maas-api-tls\") pod \"maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:29.461917 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:29.461634    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:29.940375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:29.939915    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"]
Oct 30 12:57:30.119605 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:30.119562    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7145-86b048" event={"ID":"061a8bbd-ce4a-48db-9024-f402fcc6994d","Type":"ContainerStarted","Data":"60ab80f939bd57d0e3623f0824646810000b15d0c9099a5aa699dae58ae6121a"}
Oct 30 12:57:30.124941 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:30.124899    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv" event={"ID":"e7580498-3df4-4052-b222-4020e493c364","Type":"ContainerStarted","Data":"8f1855e0ebae5b4fb73d539c61b94d92878f7f428816d241ea7166e5bfa958dc"}
Oct 30 12:57:30.158267 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:30.158197    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7145-86b048" podStartSLOduration=2.756475182 podStartE2EDuration="5.158177557s" totalImagesPullingTime="2.401702375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:30.155772127 +0000 UTC m=+1293.971098613" watchObservedRunningTime="2026-10-30 12:57:30.158177557 +0000 UTC m=+1293.973504048"
Oct 30 12:57:30.501280 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:30.501248    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:31.014430 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:31.014316    4522 patch_prober.go:29] interesting pod/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:31.014430 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:31.014407    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:31.023312 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:31.023264    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7145-86b048"]
Oct 30 12:57:31.027532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:31.027502    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7145-86b048"]
Oct 30 12:57:31.133098 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:31.133059    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv" event={"ID":"e7580498-3df4-4052-b222-4020e493c364","Type":"ContainerStarted","Data":"9893d71ac983219cb5c898c37862a4462616499f3379e519b95d11cb06fa9182"}
Oct 30 12:57:31.133276 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:31.133154    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:31.172281 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:31.172216    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv" podStartSLOduration=2.807998542 podStartE2EDuration="3.172197462s" totalImagesPullingTime="364.19892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:31.171269293 +0000 UTC m=+1294.986595793" watchObservedRunningTime="2026-10-30 12:57:31.172197462 +0000 UTC m=+1294.987523953"
Oct 30 12:57:32.141184 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:32.141136    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7145-86b048" podUID="061a8bbd-ce4a-48db-9024-f402fcc6994d" containerName="test-curl-7145-86b048" containerID="cri-o://60ab80f939bd57d0e3623f0824646810000b15d0c9099a5aa699dae58ae6121a" gracePeriod=2
Oct 30 12:57:32.154248 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:57:32.154211    4522 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7145-86b048\" is forbidden: User \"system:node:ip-10-0-129-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-133.ec2.internal' and this object" podUID="061a8bbd-ce4a-48db-9024-f402fcc6994d" pod="opendatahub/test-curl-7145-86b048"
Oct 30 12:57:32.303430 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:32.303393    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:33.149946 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:33.149914    4522 generic.go:356] "Generic (PLEG): container finished" podID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerID="2533e2550f8eb41c5a66db88615c8b11fca3bc7eae04269b6c821f1f56c22673" exitCode=0
Oct 30 12:57:33.150399 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:33.149992    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" event={"ID":"daea3195-b3c7-4500-9dfe-b753efa60e0e","Type":"ContainerDied","Data":"2533e2550f8eb41c5a66db88615c8b11fca3bc7eae04269b6c821f1f56c22673"}
Oct 30 12:57:33.150502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:33.150473    4522 scope.go:122] "RemoveContainer" containerID="2533e2550f8eb41c5a66db88615c8b11fca3bc7eae04269b6c821f1f56c22673"
Oct 30 12:57:33.152084 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:33.152055    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:33.152157 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:33.152128    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:33.152194 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:33.152163    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:33.999063 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:33.999015    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:34.187341 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:34.187308    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" event={"ID":"daea3195-b3c7-4500-9dfe-b753efa60e0e","Type":"ContainerStarted","Data":"31db14520b3ee36810b6ef7b63780e9a0919fddb3026a50a70a095bddce4ef26"}
Oct 30 12:57:34.357815 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:34.357780    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:34.457078 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:34.457036    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:34.457715 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:34.457700    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7145-86b048"
Oct 30 12:57:34.496439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:34.496400    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:57:34.496647 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:34.496606    4522 patch_prober.go:29] interesting pod/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 12:57:34.496718 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:34.496656    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 12:57:34.537889 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:34.537856    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061a8bbd-ce4a-48db-9024-f402fcc6994d-kube-api-access-hvw7l\" (UniqueName: \"kubernetes.io/projected/061a8bbd-ce4a-48db-9024-f402fcc6994d-kube-api-access-hvw7l\") pod \"061a8bbd-ce4a-48db-9024-f402fcc6994d\" (UID: \"061a8bbd-ce4a-48db-9024-f402fcc6994d\") "
Oct 30 12:57:34.539932 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:34.539900    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061a8bbd-ce4a-48db-9024-f402fcc6994d-kube-api-access-hvw7l" pod "061a8bbd-ce4a-48db-9024-f402fcc6994d" (UID: "061a8bbd-ce4a-48db-9024-f402fcc6994d"). InnerVolumeSpecName "kube-api-access-hvw7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:34.639655 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:34.639577    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvw7l\" (UniqueName: \"kubernetes.io/projected/061a8bbd-ce4a-48db-9024-f402fcc6994d-kube-api-access-hvw7l\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:35.200479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:35.200436    4522 generic.go:356] "Generic (PLEG): container finished" podID="061a8bbd-ce4a-48db-9024-f402fcc6994d" containerID="60ab80f939bd57d0e3623f0824646810000b15d0c9099a5aa699dae58ae6121a" exitCode=137
Oct 30 12:57:35.200956 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:35.200486    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7145-86b048"
Oct 30 12:57:35.200956 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:35.200523    4522 scope.go:122] "RemoveContainer" containerID="60ab80f939bd57d0e3623f0824646810000b15d0c9099a5aa699dae58ae6121a"
Oct 30 12:57:35.210339 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:35.210320    4522 scope.go:122] "RemoveContainer" containerID="60ab80f939bd57d0e3623f0824646810000b15d0c9099a5aa699dae58ae6121a"
Oct 30 12:57:35.210701 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:57:35.210680    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ab80f939bd57d0e3623f0824646810000b15d0c9099a5aa699dae58ae6121a\": container with ID starting with 60ab80f939bd57d0e3623f0824646810000b15d0c9099a5aa699dae58ae6121a not found: ID does not exist" containerID="60ab80f939bd57d0e3623f0824646810000b15d0c9099a5aa699dae58ae6121a"
Oct 30 12:57:35.210763 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:35.210709    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ab80f939bd57d0e3623f0824646810000b15d0c9099a5aa699dae58ae6121a"} err="failed to get container status \"60ab80f939bd57d0e3623f0824646810000b15d0c9099a5aa699dae58ae6121a\": rpc error: code = NotFound desc = could not find container \"60ab80f939bd57d0e3623f0824646810000b15d0c9099a5aa699dae58ae6121a\": container with ID starting with 60ab80f939bd57d0e3623f0824646810000b15d0c9099a5aa699dae58ae6121a not found: ID does not exist"
Oct 30 12:57:35.222058 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:57:35.222033    4522 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7145-86b048\" is forbidden: User \"system:node:ip-10-0-129-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-133.ec2.internal' and this object" podUID="061a8bbd-ce4a-48db-9024-f402fcc6994d" pod="opendatahub/test-curl-7145-86b048"
Oct 30 12:57:35.496178 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:35.496088    4522 patch_prober.go:29] interesting pod/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 12:57:35.496178 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:35.496150    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 12:57:35.787374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:35.785716    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:36.335767 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:36.335734    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061a8bbd-ce4a-48db-9024-f402fcc6994d" path="/var/lib/kubelet/pods/061a8bbd-ce4a-48db-9024-f402fcc6994d/volumes"
Oct 30 12:57:36.343909 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:57:36.343872    4522 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7145-86b048\" is forbidden: User \"system:node:ip-10-0-129-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-133.ec2.internal' and this object" podUID="061a8bbd-ce4a-48db-9024-f402fcc6994d" pod="opendatahub/test-curl-7145-86b048"
Oct 30 12:57:36.496052 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:36.496021    4522 patch_prober.go:29] interesting pod/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 12:57:36.496226 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:36.496067    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 12:57:36.921947 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:36.921870    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"]
Oct 30 12:57:36.951295 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:36.951266    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"]
Oct 30 12:57:37.048066 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:37.048031    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:37.168500 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:37.153109    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:57:37.414921 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:37.414859    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:37.495973 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:37.495940    4522 patch_prober.go:29] interesting pod/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 12:57:37.496216 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:37.496192    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 12:57:37.542804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:37.542770    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:38.106162 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:38.106129    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"
Oct 30 12:57:38.496232 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:38.496118    4522 patch_prober.go:29] interesting pod/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 12:57:38.496232 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:38.496173    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 12:57:39.319472 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:39.319439    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:39.503401 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:39.498127    4522 patch_prober.go:29] interesting pod/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 12:57:39.503401 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:39.498193    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 12:57:40.500911 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:40.500882    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:57:40.501198 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:40.501176    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:57:40.501847 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:40.501827    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:57:41.084605 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:41.084575    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:41.922380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:41.922290    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk" podUID="7237c611-57be-484a-b096-0e12164ab8eb" containerName="payload-processing" containerID="cri-o://cf3bbe7d668e5b5aa27a6ce6e71f95e57da90445d306c902228010a4706882bf" gracePeriod=25
Oct 30 12:57:41.951545 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:41.951510    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59" podUID="8e969a4e-090b-4931-a158-8b45c1c29eb2" containerName="payload-pre-processing" containerID="cri-o://c475d060e0c500fe804dd953641ac2b9bc1afc3f7e92108bb7a7ea6517ed4885" gracePeriod=25
Oct 30 12:57:42.037005 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.036974    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"]
Oct 30 12:57:42.037652 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.037636    4522 cpu_manager.go:414] "containerMap: removing container" podUID="061a8bbd-ce4a-48db-9024-f402fcc6994d" containerName="test-curl-7145-86b048"
Oct 30 12:57:42.037714 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.037653    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061a8bbd-ce4a-48db-9024-f402fcc6994d" containerName="test-curl-7145-86b048"
Oct 30 12:57:42.037753 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.037744    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="061a8bbd-ce4a-48db-9024-f402fcc6994d" containerName="test-curl-7145-86b048"
Oct 30 12:57:42.041676 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.041660    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.062315 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.062286    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"]
Oct 30 12:57:42.101505 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.101448    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"]
Oct 30 12:57:42.129037 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.129003    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"]
Oct 30 12:57:42.157993 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.157283    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.157993 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.157437    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.157993 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.157572    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68996c3-3081-47c8-9878-d2e438a298a7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.157993 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.157626    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.157993 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.157677    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.157993 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.157709    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.157993 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.157770    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mc4j\" (UniqueName: \"kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-kube-api-access-5mc4j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.229379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.229318    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:42.258875 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.258843    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.259007 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.258931    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.259007 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.258970    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.259150 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.259036    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mc4j\" (UniqueName: \"kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-kube-api-access-5mc4j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.259150 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.259118    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.259309 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.259219    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.259403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.259348    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68996c3-3081-47c8-9878-d2e438a298a7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.259738 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.259709    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.262549 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.262134    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.262549 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.262475    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.263754 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.263735    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.264064 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.264045    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68996c3-3081-47c8-9878-d2e438a298a7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.281085 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.281053    4522 generic.go:356] "Generic (PLEG): container finished" podID="8e969a4e-090b-4931-a158-8b45c1c29eb2" containerID="c475d060e0c500fe804dd953641ac2b9bc1afc3f7e92108bb7a7ea6517ed4885" exitCode=0
Oct 30 12:57:42.281263 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.281239    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59" event={"ID":"8e969a4e-090b-4931-a158-8b45c1c29eb2","Type":"ContainerDied","Data":"c475d060e0c500fe804dd953641ac2b9bc1afc3f7e92108bb7a7ea6517ed4885"}
Oct 30 12:57:42.283003 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.282979    4522 generic.go:356] "Generic (PLEG): container finished" podID="7237c611-57be-484a-b096-0e12164ab8eb" containerID="cf3bbe7d668e5b5aa27a6ce6e71f95e57da90445d306c902228010a4706882bf" exitCode=0
Oct 30 12:57:42.283733 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.283452    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk" event={"ID":"7237c611-57be-484a-b096-0e12164ab8eb","Type":"ContainerDied","Data":"cf3bbe7d668e5b5aa27a6ce6e71f95e57da90445d306c902228010a4706882bf"}
Oct 30 12:57:42.283733 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.283511    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk" event={"ID":"7237c611-57be-484a-b096-0e12164ab8eb","Type":"ContainerDied","Data":"116a777b51f2736b18addcb0b3c908a28a25970e144b1aed695869920e6a6107"}
Oct 30 12:57:42.283733 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.283537    4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="116a777b51f2736b18addcb0b3c908a28a25970e144b1aed695869920e6a6107"
Oct 30 12:57:42.283733 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.283576    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mc4j\" (UniqueName: \"kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-kube-api-access-5mc4j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.283733 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.283657    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gjccx\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.294769 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.294751    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"
Oct 30 12:57:42.297640 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.297621    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"
Oct 30 12:57:42.403038 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.401635    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:42.403038 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.402724    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:42.476403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.474893    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e969a4e-090b-4931-a158-8b45c1c29eb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e969a4e-090b-4931-a158-8b45c1c29eb2-plugins-config-volume\") pod \"8e969a4e-090b-4931-a158-8b45c1c29eb2\" (UID: \"8e969a4e-090b-4931-a158-8b45c1c29eb2\") "
Oct 30 12:57:42.476403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.475158    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7237c611-57be-484a-b096-0e12164ab8eb-kube-api-access-rph9x\" (UniqueName: \"kubernetes.io/projected/7237c611-57be-484a-b096-0e12164ab8eb-kube-api-access-rph9x\") pod \"7237c611-57be-484a-b096-0e12164ab8eb\" (UID: \"7237c611-57be-484a-b096-0e12164ab8eb\") "
Oct 30 12:57:42.476403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.475398    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e969a4e-090b-4931-a158-8b45c1c29eb2-kube-api-access-w277s\" (UniqueName: \"kubernetes.io/projected/8e969a4e-090b-4931-a158-8b45c1c29eb2-kube-api-access-w277s\") pod \"8e969a4e-090b-4931-a158-8b45c1c29eb2\" (UID: \"8e969a4e-090b-4931-a158-8b45c1c29eb2\") "
Oct 30 12:57:42.476403 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.475472    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7237c611-57be-484a-b096-0e12164ab8eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7237c611-57be-484a-b096-0e12164ab8eb-plugins-config-volume\") pod \"7237c611-57be-484a-b096-0e12164ab8eb\" (UID: \"7237c611-57be-484a-b096-0e12164ab8eb\") "
Oct 30 12:57:42.515394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.515326    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7237c611-57be-484a-b096-0e12164ab8eb-plugins-config-volume" pod "7237c611-57be-484a-b096-0e12164ab8eb" (UID: "7237c611-57be-484a-b096-0e12164ab8eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:57:42.518011 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.517971    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e969a4e-090b-4931-a158-8b45c1c29eb2-plugins-config-volume" pod "8e969a4e-090b-4931-a158-8b45c1c29eb2" (UID: "8e969a4e-090b-4931-a158-8b45c1c29eb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:57:42.531251 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.528010    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7237c611-57be-484a-b096-0e12164ab8eb-kube-api-access-rph9x" pod "7237c611-57be-484a-b096-0e12164ab8eb" (UID: "7237c611-57be-484a-b096-0e12164ab8eb"). InnerVolumeSpecName "kube-api-access-rph9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:42.531251 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.530001    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e969a4e-090b-4931-a158-8b45c1c29eb2-kube-api-access-w277s" pod "8e969a4e-090b-4931-a158-8b45c1c29eb2" (UID: "8e969a4e-090b-4931-a158-8b45c1c29eb2"). InnerVolumeSpecName "kube-api-access-w277s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:42.615725 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.615685    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w277s\" (UniqueName: \"kubernetes.io/projected/8e969a4e-090b-4931-a158-8b45c1c29eb2-kube-api-access-w277s\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:42.615725 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.615726    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7237c611-57be-484a-b096-0e12164ab8eb-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:42.615915 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.615741    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e969a4e-090b-4931-a158-8b45c1c29eb2-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:42.615915 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.615755    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rph9x\" (UniqueName: \"kubernetes.io/projected/7237c611-57be-484a-b096-0e12164ab8eb-kube-api-access-rph9x\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:57:42.961381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.928924    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:43.024439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:42.986137    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"]
Oct 30 12:57:43.128455 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:43.121054    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:43.128455 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:43.121144    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:43.135298 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:43.134460    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:43.328186 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:43.323680    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:43.333133 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:43.333096    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx" event={"ID":"b68996c3-3081-47c8-9878-d2e438a298a7","Type":"ContainerStarted","Data":"262e594f140924e795250beea43a377762152d7bd0ae788c7ec987715bb0e353"}
Oct 30 12:57:43.335612 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:43.334452    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"
Oct 30 12:57:43.335612 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:43.334481    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"
Oct 30 12:57:43.335612 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:43.334518    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59" event={"ID":"8e969a4e-090b-4931-a158-8b45c1c29eb2","Type":"ContainerDied","Data":"53b92e78be2a5a3d3deb79bdb60821c98170cdfca8be7c4a37c10ef91d470fc3"}
Oct 30 12:57:43.335612 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:43.334562    4522 scope.go:122] "RemoveContainer" containerID="c475d060e0c500fe804dd953641ac2b9bc1afc3f7e92108bb7a7ea6517ed4885"
Oct 30 12:57:43.412198 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:43.412164    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"]
Oct 30 12:57:43.415463 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:43.415433    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a4499-547794fbb6-ttvxk"]
Oct 30 12:57:43.446070 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:43.446038    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"]
Oct 30 12:57:43.452490 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:43.452463    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a4499-565465d8dc-jzw59"]
Oct 30 12:57:44.336265 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:44.336231    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7237c611-57be-484a-b096-0e12164ab8eb" path="/var/lib/kubelet/pods/7237c611-57be-484a-b096-0e12164ab8eb/volumes"
Oct 30 12:57:44.336868 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:44.336842    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e969a4e-090b-4931-a158-8b45c1c29eb2" path="/var/lib/kubelet/pods/8e969a4e-090b-4931-a158-8b45c1c29eb2/volumes"
Oct 30 12:57:44.343728 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:44.343703    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx" event={"ID":"b68996c3-3081-47c8-9878-d2e438a298a7","Type":"ContainerStarted","Data":"9465fccc24422763da2a1c7e03c8e8f8be411b6b7c10d5069c0595361263c6c1"}
Oct 30 12:57:44.344525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:44.344501    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:44.346656 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:44.346635    4522 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-gjccx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:57:44.346787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:44.346680    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx" podUID="b68996c3-3081-47c8-9878-d2e438a298a7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:57:44.401097 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:44.401046    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx" podStartSLOduration=2.401032483 podStartE2EDuration="2.401032483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:44.39607138 +0000 UTC m=+1308.211397870" watchObservedRunningTime="2026-10-30 12:57:44.401032483 +0000 UTC m=+1308.216358972"
Oct 30 12:57:44.873951 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:44.873919    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:45.350769 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:45.350737    4522 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-gjccx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:57:45.351290 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:45.350787    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx" podUID="b68996c3-3081-47c8-9878-d2e438a298a7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:57:46.357968 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:46.357938    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 12:57:46.578236 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:46.578197    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:47.129594 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:47.129528    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq" podUID="c4fd5c1b-b147-44be-b82a-a69fc5f2ee11" containerName="payload-pre-processing" containerID="cri-o://d100541ef2f5aaae7fdff75adb3999203890dc44e019981d9c619b9280767a1f" gracePeriod=25
Oct 30 12:57:47.283623 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:47.283564    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8" podUID="08206131-6355-4479-bfaf-a9920a03bc91" containerName="payload-processing" containerID="cri-o://31e39cd9cd0badefc6d7c0c8255ac25662b933a7afc43a5f50289ae81590c61b" gracePeriod=25
Oct 30 12:57:48.439480 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:48.439445    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:48.540597 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:48.540564    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:50.155792 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:50.155758    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:50.301843 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:50.301784    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:51.133903 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:57:51.133858    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:57:52.240171 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:52.240139    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:53.641556 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:53.641521    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8" podUID="08206131-6355-4479-bfaf-a9920a03bc91" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:57:53.641556 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:53.641546    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq" podUID="c4fd5c1b-b147-44be-b82a-a69fc5f2ee11" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:57:53.950950 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:53.950855    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:55.623791 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:55.623750    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:57.305435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:57.305399    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:58.938628 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:57:58.938591    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:00.389367 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:00.389320    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"]
Oct 30 12:58:00.389813 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:00.389552    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9" podUID="c566749d-af52-4992-803c-334d8e0d08c6" containerName="maas-api" containerID="cri-o://6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287" gracePeriod=30
Oct 30 12:58:00.444406 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:00.444346    4522 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc566749d_af52_4992_803c_334d8e0d08c6.slice/crio-6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 12:58:00.452618 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:00.452587    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:00.856099 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:00.852392    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:58:00.948391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:00.943023    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c566749d-af52-4992-803c-334d8e0d08c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c566749d-af52-4992-803c-334d8e0d08c6-openshift-service-ca\") pod \"c566749d-af52-4992-803c-334d8e0d08c6\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") "
Oct 30 12:58:00.948391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:00.943103    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-metrics-certs\") pod \"c566749d-af52-4992-803c-334d8e0d08c6\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") "
Oct 30 12:58:00.948391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:00.943188    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-maas-api-tls\") pod \"c566749d-af52-4992-803c-334d8e0d08c6\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") "
Oct 30 12:58:00.948391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:00.943249    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c566749d-af52-4992-803c-334d8e0d08c6-kube-api-access-4r282\" (UniqueName: \"kubernetes.io/projected/c566749d-af52-4992-803c-334d8e0d08c6-kube-api-access-4r282\") pod \"c566749d-af52-4992-803c-334d8e0d08c6\" (UID: \"c566749d-af52-4992-803c-334d8e0d08c6\") "
Oct 30 12:58:00.949502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:00.949159    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c566749d-af52-4992-803c-334d8e0d08c6-openshift-service-ca" pod "c566749d-af52-4992-803c-334d8e0d08c6" (UID: "c566749d-af52-4992-803c-334d8e0d08c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:01.005639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:00.971497    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c566749d-af52-4992-803c-334d8e0d08c6-kube-api-access-4r282" pod "c566749d-af52-4992-803c-334d8e0d08c6" (UID: "c566749d-af52-4992-803c-334d8e0d08c6"). InnerVolumeSpecName "kube-api-access-4r282". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:01.005639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:00.974078    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-maas-api-tls" pod "c566749d-af52-4992-803c-334d8e0d08c6" (UID: "c566749d-af52-4992-803c-334d8e0d08c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:01.005639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:00.974184    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-metrics-certs" pod "c566749d-af52-4992-803c-334d8e0d08c6" (UID: "c566749d-af52-4992-803c-334d8e0d08c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:01.068389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.049707    4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c566749d-af52-4992-803c-334d8e0d08c6-openshift-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:01.068389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.049751    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:01.068389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.049768    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c566749d-af52-4992-803c-334d8e0d08c6-maas-api-tls\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:01.068389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.049782    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r282\" (UniqueName: \"kubernetes.io/projected/c566749d-af52-4992-803c-334d8e0d08c6-kube-api-access-4r282\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:01.281947 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.281904    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:01.384013 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.383965    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:01.483413 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.483345    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:01.613436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.606083    4522 generic.go:356] "Generic (PLEG): container finished" podID="c566749d-af52-4992-803c-334d8e0d08c6" containerID="6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287" exitCode=0
Oct 30 12:58:01.613436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.606213    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9" event={"ID":"c566749d-af52-4992-803c-334d8e0d08c6","Type":"ContainerDied","Data":"6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287"}
Oct 30 12:58:01.613436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.606262    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9" event={"ID":"c566749d-af52-4992-803c-334d8e0d08c6","Type":"ContainerDied","Data":"4f0ccb08c7ed196c0bea68124b705982f2484fc8f23c4445a928537e5004efc8"}
Oct 30 12:58:01.613436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.606289    4522 scope.go:122] "RemoveContainer" containerID="6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287"
Oct 30 12:58:01.613436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.606457    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"
Oct 30 12:58:01.648092 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.648070    4522 scope.go:122] "RemoveContainer" containerID="6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287"
Oct 30 12:58:01.648481 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:01.648460    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287\": container with ID starting with 6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287 not found: ID does not exist" containerID="6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287"
Oct 30 12:58:01.648568 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.648491    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287"} err="failed to get container status \"6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287\": rpc error: code = NotFound desc = could not find container \"6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287\": container with ID starting with 6caa5e8d5fde1c201fb6b2b77183ee450ea556408fc4bfc40c661ba9f6fce287 not found: ID does not exist"
Oct 30 12:58:01.680173 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.680141    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"]
Oct 30 12:58:01.691746 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.691717    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a4499-cdc5998fd-zgzj9"]
Oct 30 12:58:01.961091 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:01.961000    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:02.174641 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:02.174607    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"]
Oct 30 12:58:02.197947 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:02.197912    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"]
Oct 30 12:58:02.299267 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:02.299215    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:02.334801 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:02.334768    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c566749d-af52-4992-803c-334d8e0d08c6" path="/var/lib/kubelet/pods/c566749d-af52-4992-803c-334d8e0d08c6/volumes"
Oct 30 12:58:02.428376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:02.416925    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:02.781949 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:02.781910    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:03.160905 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:03.160812    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"]
Oct 30 12:58:03.249519 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:03.249477    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:03.349736 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:03.349697    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:03.476438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:03.473336    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:03.665084 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:03.665044    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8" podUID="08206131-6355-4479-bfaf-a9920a03bc91" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:58:03.666268 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:03.666236    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq" podUID="c4fd5c1b-b147-44be-b82a-a69fc5f2ee11" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:58:04.914457 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:04.914420    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:05.556376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:05.555405    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:05.768948 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:05.768910    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:05.958044 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:05.957940    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:07.174979 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.174946    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq" podUID="53cab76d-b312-4afb-bc3d-c29ece805ce1" containerName="payload-processing" containerID="cri-o://0506fd950c009adb542f7ccb96adb95d9d155e214fc26d886d0f8b384f886c90" gracePeriod=25
Oct 30 12:58:07.198570 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.198519    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g" podUID="917fa195-0b4a-4cb8-8021-0573bcae153b" containerName="payload-pre-processing" containerID="cri-o://d6a49bd9b1c0b2f98032c633d2b431c04e8adfca089d2c4219ba45e7f0ef0ac5" gracePeriod=25
Oct 30 12:58:07.516307 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.516283    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"
Oct 30 12:58:07.528239 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.528214    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"
Oct 30 12:58:07.615144 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.615113    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"
Oct 30 12:58:07.645169 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.645108    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"
Oct 30 12:58:07.690041 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.690013    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cab76d-b312-4afb-bc3d-c29ece805ce1-kube-api-access-5jksw\" (UniqueName: \"kubernetes.io/projected/53cab76d-b312-4afb-bc3d-c29ece805ce1-kube-api-access-5jksw\") pod \"53cab76d-b312-4afb-bc3d-c29ece805ce1\" (UID: \"53cab76d-b312-4afb-bc3d-c29ece805ce1\") "
Oct 30 12:58:07.690228 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.690209    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53cab76d-b312-4afb-bc3d-c29ece805ce1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53cab76d-b312-4afb-bc3d-c29ece805ce1-plugins-config-volume\") pod \"53cab76d-b312-4afb-bc3d-c29ece805ce1\" (UID: \"53cab76d-b312-4afb-bc3d-c29ece805ce1\") "
Oct 30 12:58:07.690295 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.690253    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-kube-api-access-4t24j\" (UniqueName: \"kubernetes.io/projected/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-kube-api-access-4t24j\") pod \"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11\" (UID: \"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11\") "
Oct 30 12:58:07.690452 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.690430    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-plugins-config-volume\") pod \"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11\" (UID: \"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11\") "
Oct 30 12:58:07.690805 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.690660    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cab76d-b312-4afb-bc3d-c29ece805ce1-plugins-config-volume" pod "53cab76d-b312-4afb-bc3d-c29ece805ce1" (UID: "53cab76d-b312-4afb-bc3d-c29ece805ce1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:07.690916 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.690887    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-plugins-config-volume" pod "c4fd5c1b-b147-44be-b82a-a69fc5f2ee11" (UID: "c4fd5c1b-b147-44be-b82a-a69fc5f2ee11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:07.691151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.691089    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53cab76d-b312-4afb-bc3d-c29ece805ce1-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:07.691151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.691112    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:07.693059 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.693031    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cab76d-b312-4afb-bc3d-c29ece805ce1-kube-api-access-5jksw" pod "53cab76d-b312-4afb-bc3d-c29ece805ce1" (UID: "53cab76d-b312-4afb-bc3d-c29ece805ce1"). InnerVolumeSpecName "kube-api-access-5jksw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:07.694429 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.694400    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-kube-api-access-4t24j" pod "c4fd5c1b-b147-44be-b82a-a69fc5f2ee11" (UID: "c4fd5c1b-b147-44be-b82a-a69fc5f2ee11"). InnerVolumeSpecName "kube-api-access-4t24j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:07.699968 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.699937    4522 generic.go:356] "Generic (PLEG): container finished" podID="917fa195-0b4a-4cb8-8021-0573bcae153b" containerID="d6a49bd9b1c0b2f98032c633d2b431c04e8adfca089d2c4219ba45e7f0ef0ac5" exitCode=0
Oct 30 12:58:07.700075 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.699971    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"
Oct 30 12:58:07.700165 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.700140    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g" event={"ID":"917fa195-0b4a-4cb8-8021-0573bcae153b","Type":"ContainerDied","Data":"d6a49bd9b1c0b2f98032c633d2b431c04e8adfca089d2c4219ba45e7f0ef0ac5"}
Oct 30 12:58:07.700229 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.700198    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g" event={"ID":"917fa195-0b4a-4cb8-8021-0573bcae153b","Type":"ContainerDied","Data":"60fe0af1f7a12b3381469ea6d48a33748da0e2ce643e935bef6d6e926bb79382"}
Oct 30 12:58:07.700285 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.700235    4522 scope.go:122] "RemoveContainer" containerID="d6a49bd9b1c0b2f98032c633d2b431c04e8adfca089d2c4219ba45e7f0ef0ac5"
Oct 30 12:58:07.702458 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.702343    4522 generic.go:356] "Generic (PLEG): container finished" podID="08206131-6355-4479-bfaf-a9920a03bc91" containerID="31e39cd9cd0badefc6d7c0c8255ac25662b933a7afc43a5f50289ae81590c61b" exitCode=0
Oct 30 12:58:07.702458 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.702406    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"
Oct 30 12:58:07.702620 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.702522    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8" event={"ID":"08206131-6355-4479-bfaf-a9920a03bc91","Type":"ContainerDied","Data":"31e39cd9cd0badefc6d7c0c8255ac25662b933a7afc43a5f50289ae81590c61b"}
Oct 30 12:58:07.702620 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.702565    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8" event={"ID":"08206131-6355-4479-bfaf-a9920a03bc91","Type":"ContainerDied","Data":"2edbe9bf6bc479d605701e7d2d141d2a0abf79ec541d2cbc2a3e8be288325012"}
Oct 30 12:58:07.704624 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.704587    4522 generic.go:356] "Generic (PLEG): container finished" podID="c4fd5c1b-b147-44be-b82a-a69fc5f2ee11" containerID="d100541ef2f5aaae7fdff75adb3999203890dc44e019981d9c619b9280767a1f" exitCode=0
Oct 30 12:58:07.704770 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.704637    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"
Oct 30 12:58:07.704770 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.704707    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq" event={"ID":"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11","Type":"ContainerDied","Data":"d100541ef2f5aaae7fdff75adb3999203890dc44e019981d9c619b9280767a1f"}
Oct 30 12:58:07.704770 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.704748    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq" event={"ID":"c4fd5c1b-b147-44be-b82a-a69fc5f2ee11","Type":"ContainerDied","Data":"25c35e703082eaca6b6a654e867a7b0864773d4664a28662928099ec7bf49378"}
Oct 30 12:58:07.706467 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.706438    4522 generic.go:356] "Generic (PLEG): container finished" podID="53cab76d-b312-4afb-bc3d-c29ece805ce1" containerID="0506fd950c009adb542f7ccb96adb95d9d155e214fc26d886d0f8b384f886c90" exitCode=0
Oct 30 12:58:07.706553 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.706519    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq" event={"ID":"53cab76d-b312-4afb-bc3d-c29ece805ce1","Type":"ContainerDied","Data":"0506fd950c009adb542f7ccb96adb95d9d155e214fc26d886d0f8b384f886c90"}
Oct 30 12:58:07.706597 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.706555    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq" event={"ID":"53cab76d-b312-4afb-bc3d-c29ece805ce1","Type":"ContainerDied","Data":"f949b4cb0eeaf5571faf26bd7c3e102ae2d393a9fed764a1a24ee8c416fd9fa3"}
Oct 30 12:58:07.706638 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.706619    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"
Oct 30 12:58:07.714247 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.714227    4522 scope.go:122] "RemoveContainer" containerID="d6a49bd9b1c0b2f98032c633d2b431c04e8adfca089d2c4219ba45e7f0ef0ac5"
Oct 30 12:58:07.714893 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:07.714843    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a49bd9b1c0b2f98032c633d2b431c04e8adfca089d2c4219ba45e7f0ef0ac5\": container with ID starting with d6a49bd9b1c0b2f98032c633d2b431c04e8adfca089d2c4219ba45e7f0ef0ac5 not found: ID does not exist" containerID="d6a49bd9b1c0b2f98032c633d2b431c04e8adfca089d2c4219ba45e7f0ef0ac5"
Oct 30 12:58:07.714981 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.714881    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a49bd9b1c0b2f98032c633d2b431c04e8adfca089d2c4219ba45e7f0ef0ac5"} err="failed to get container status \"d6a49bd9b1c0b2f98032c633d2b431c04e8adfca089d2c4219ba45e7f0ef0ac5\": rpc error: code = NotFound desc = could not find container \"d6a49bd9b1c0b2f98032c633d2b431c04e8adfca089d2c4219ba45e7f0ef0ac5\": container with ID starting with d6a49bd9b1c0b2f98032c633d2b431c04e8adfca089d2c4219ba45e7f0ef0ac5 not found: ID does not exist"
Oct 30 12:58:07.714981 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.714906    4522 scope.go:122] "RemoveContainer" containerID="31e39cd9cd0badefc6d7c0c8255ac25662b933a7afc43a5f50289ae81590c61b"
Oct 30 12:58:07.726474 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.726456    4522 scope.go:122] "RemoveContainer" containerID="31e39cd9cd0badefc6d7c0c8255ac25662b933a7afc43a5f50289ae81590c61b"
Oct 30 12:58:07.726811 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:07.726787    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e39cd9cd0badefc6d7c0c8255ac25662b933a7afc43a5f50289ae81590c61b\": container with ID starting with 31e39cd9cd0badefc6d7c0c8255ac25662b933a7afc43a5f50289ae81590c61b not found: ID does not exist" containerID="31e39cd9cd0badefc6d7c0c8255ac25662b933a7afc43a5f50289ae81590c61b"
Oct 30 12:58:07.726865 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.726821    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e39cd9cd0badefc6d7c0c8255ac25662b933a7afc43a5f50289ae81590c61b"} err="failed to get container status \"31e39cd9cd0badefc6d7c0c8255ac25662b933a7afc43a5f50289ae81590c61b\": rpc error: code = NotFound desc = could not find container \"31e39cd9cd0badefc6d7c0c8255ac25662b933a7afc43a5f50289ae81590c61b\": container with ID starting with 31e39cd9cd0badefc6d7c0c8255ac25662b933a7afc43a5f50289ae81590c61b not found: ID does not exist"
Oct 30 12:58:07.726865 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.726842    4522 scope.go:122] "RemoveContainer" containerID="d100541ef2f5aaae7fdff75adb3999203890dc44e019981d9c619b9280767a1f"
Oct 30 12:58:07.737545 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.737512    4522 scope.go:122] "RemoveContainer" containerID="d100541ef2f5aaae7fdff75adb3999203890dc44e019981d9c619b9280767a1f"
Oct 30 12:58:07.737883 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:07.737845    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d100541ef2f5aaae7fdff75adb3999203890dc44e019981d9c619b9280767a1f\": container with ID starting with d100541ef2f5aaae7fdff75adb3999203890dc44e019981d9c619b9280767a1f not found: ID does not exist" containerID="d100541ef2f5aaae7fdff75adb3999203890dc44e019981d9c619b9280767a1f"
Oct 30 12:58:07.737983 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.737887    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d100541ef2f5aaae7fdff75adb3999203890dc44e019981d9c619b9280767a1f"} err="failed to get container status \"d100541ef2f5aaae7fdff75adb3999203890dc44e019981d9c619b9280767a1f\": rpc error: code = NotFound desc = could not find container \"d100541ef2f5aaae7fdff75adb3999203890dc44e019981d9c619b9280767a1f\": container with ID starting with d100541ef2f5aaae7fdff75adb3999203890dc44e019981d9c619b9280767a1f not found: ID does not exist"
Oct 30 12:58:07.737983 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.737903    4522 scope.go:122] "RemoveContainer" containerID="0506fd950c009adb542f7ccb96adb95d9d155e214fc26d886d0f8b384f886c90"
Oct 30 12:58:07.747456 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.747440    4522 scope.go:122] "RemoveContainer" containerID="0506fd950c009adb542f7ccb96adb95d9d155e214fc26d886d0f8b384f886c90"
Oct 30 12:58:07.747687 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:07.747667    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0506fd950c009adb542f7ccb96adb95d9d155e214fc26d886d0f8b384f886c90\": container with ID starting with 0506fd950c009adb542f7ccb96adb95d9d155e214fc26d886d0f8b384f886c90 not found: ID does not exist" containerID="0506fd950c009adb542f7ccb96adb95d9d155e214fc26d886d0f8b384f886c90"
Oct 30 12:58:07.747737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.747699    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0506fd950c009adb542f7ccb96adb95d9d155e214fc26d886d0f8b384f886c90"} err="failed to get container status \"0506fd950c009adb542f7ccb96adb95d9d155e214fc26d886d0f8b384f886c90\": rpc error: code = NotFound desc = could not find container \"0506fd950c009adb542f7ccb96adb95d9d155e214fc26d886d0f8b384f886c90\": container with ID starting with 0506fd950c009adb542f7ccb96adb95d9d155e214fc26d886d0f8b384f886c90 not found: ID does not exist"
Oct 30 12:58:07.792091 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.792068    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/917fa195-0b4a-4cb8-8021-0573bcae153b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/917fa195-0b4a-4cb8-8021-0573bcae153b-plugins-config-volume\") pod \"917fa195-0b4a-4cb8-8021-0573bcae153b\" (UID: \"917fa195-0b4a-4cb8-8021-0573bcae153b\") "
Oct 30 12:58:07.792221 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.792208    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08206131-6355-4479-bfaf-a9920a03bc91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08206131-6355-4479-bfaf-a9920a03bc91-plugins-config-volume\") pod \"08206131-6355-4479-bfaf-a9920a03bc91\" (UID: \"08206131-6355-4479-bfaf-a9920a03bc91\") "
Oct 30 12:58:07.792309 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.792294    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/917fa195-0b4a-4cb8-8021-0573bcae153b-kube-api-access-zk8nc\" (UniqueName: \"kubernetes.io/projected/917fa195-0b4a-4cb8-8021-0573bcae153b-kube-api-access-zk8nc\") pod \"917fa195-0b4a-4cb8-8021-0573bcae153b\" (UID: \"917fa195-0b4a-4cb8-8021-0573bcae153b\") "
Oct 30 12:58:07.792385 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.792346    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08206131-6355-4479-bfaf-a9920a03bc91-kube-api-access-tngtp\" (UniqueName: \"kubernetes.io/projected/08206131-6355-4479-bfaf-a9920a03bc91-kube-api-access-tngtp\") pod \"08206131-6355-4479-bfaf-a9920a03bc91\" (UID: \"08206131-6355-4479-bfaf-a9920a03bc91\") "
Oct 30 12:58:07.792654 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.792629    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08206131-6355-4479-bfaf-a9920a03bc91-plugins-config-volume" pod "08206131-6355-4479-bfaf-a9920a03bc91" (UID: "08206131-6355-4479-bfaf-a9920a03bc91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:07.792769 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.792602    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917fa195-0b4a-4cb8-8021-0573bcae153b-plugins-config-volume" pod "917fa195-0b4a-4cb8-8021-0573bcae153b" (UID: "917fa195-0b4a-4cb8-8021-0573bcae153b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:07.793062 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.793044    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t24j\" (UniqueName: \"kubernetes.io/projected/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11-kube-api-access-4t24j\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:07.793148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.793064    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08206131-6355-4479-bfaf-a9920a03bc91-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:07.793148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.793074    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jksw\" (UniqueName: \"kubernetes.io/projected/53cab76d-b312-4afb-bc3d-c29ece805ce1-kube-api-access-5jksw\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:07.793148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.793083    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/917fa195-0b4a-4cb8-8021-0573bcae153b-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:07.794858 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.794837    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917fa195-0b4a-4cb8-8021-0573bcae153b-kube-api-access-zk8nc" pod "917fa195-0b4a-4cb8-8021-0573bcae153b" (UID: "917fa195-0b4a-4cb8-8021-0573bcae153b"). InnerVolumeSpecName "kube-api-access-zk8nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:07.794953 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.794868    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08206131-6355-4479-bfaf-a9920a03bc91-kube-api-access-tngtp" pod "08206131-6355-4479-bfaf-a9920a03bc91" (UID: "08206131-6355-4479-bfaf-a9920a03bc91"). InnerVolumeSpecName "kube-api-access-tngtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:07.797756 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.797733    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"]
Oct 30 12:58:07.804073 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.804051    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef1186-7f6d9c4844-qvjlq"]
Oct 30 12:58:07.834915 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.834893    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"]
Oct 30 12:58:07.840555 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.840536    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a04e002-849d65ff6c-xhfhq"]
Oct 30 12:58:07.850054 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.850030    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:07.901639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.899301    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk8nc\" (UniqueName: \"kubernetes.io/projected/917fa195-0b4a-4cb8-8021-0573bcae153b-kube-api-access-zk8nc\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:07.901639 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:07.899346    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tngtp\" (UniqueName: \"kubernetes.io/projected/08206131-6355-4479-bfaf-a9920a03bc91-kube-api-access-tngtp\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:08.049904 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.049868    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"]
Oct 30 12:58:08.053427 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.053401    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-nhql8"]
Oct 30 12:58:08.087001 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.086969    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"]
Oct 30 12:58:08.089954 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.089935    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef1186-85dd565675-phv7g"]
Oct 30 12:58:08.161799 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.161713    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn" podUID="5fb623f7-f001-4d88-b898-4e50b421b05b" containerName="payload-pre-processing" containerID="cri-o://6416ca5e28d5c38cd725e39b0c0fcd60d6b25ea3b0f2266ae3cb7b3026b57b6d" gracePeriod=25
Oct 30 12:58:08.214721 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.214688    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:08.316137 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.315585    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:08.383272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.375612    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08206131-6355-4479-bfaf-a9920a03bc91" path="/var/lib/kubelet/pods/08206131-6355-4479-bfaf-a9920a03bc91/volumes"
Oct 30 12:58:08.383272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.376473    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53cab76d-b312-4afb-bc3d-c29ece805ce1" path="/var/lib/kubelet/pods/53cab76d-b312-4afb-bc3d-c29ece805ce1/volumes"
Oct 30 12:58:08.383272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.376943    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="917fa195-0b4a-4cb8-8021-0573bcae153b" path="/var/lib/kubelet/pods/917fa195-0b4a-4cb8-8021-0573bcae153b/volumes"
Oct 30 12:58:08.383272 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.377334    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fd5c1b-b147-44be-b82a-a69fc5f2ee11" path="/var/lib/kubelet/pods/c4fd5c1b-b147-44be-b82a-a69fc5f2ee11/volumes"
Oct 30 12:58:08.533983 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.533951    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"]
Oct 30 12:58:08.534990 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.534954    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy" containerID="cri-o://31db14520b3ee36810b6ef7b63780e9a0919fddb3026a50a70a095bddce4ef26" gracePeriod=30
Oct 30 12:58:08.545968 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.545938    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"
Oct 30 12:58:08.554722 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.554689    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:08.775279 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.745019    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb623f7-f001-4d88-b898-4e50b421b05b-kube-api-access-zqxj6\" (UniqueName: \"kubernetes.io/projected/5fb623f7-f001-4d88-b898-4e50b421b05b-kube-api-access-zqxj6\") pod \"5fb623f7-f001-4d88-b898-4e50b421b05b\" (UID: \"5fb623f7-f001-4d88-b898-4e50b421b05b\") "
Oct 30 12:58:08.775279 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.745183    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fb623f7-f001-4d88-b898-4e50b421b05b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb623f7-f001-4d88-b898-4e50b421b05b-plugins-config-volume\") pod \"5fb623f7-f001-4d88-b898-4e50b421b05b\" (UID: \"5fb623f7-f001-4d88-b898-4e50b421b05b\") "
Oct 30 12:58:08.775279 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.757593    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb623f7-f001-4d88-b898-4e50b421b05b-plugins-config-volume" pod "5fb623f7-f001-4d88-b898-4e50b421b05b" (UID: "5fb623f7-f001-4d88-b898-4e50b421b05b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:08.830241 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.830191    4522 generic.go:356] "Generic (PLEG): container finished" podID="5fb623f7-f001-4d88-b898-4e50b421b05b" containerID="6416ca5e28d5c38cd725e39b0c0fcd60d6b25ea3b0f2266ae3cb7b3026b57b6d" exitCode=0
Oct 30 12:58:08.830468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.830381    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn" event={"ID":"5fb623f7-f001-4d88-b898-4e50b421b05b","Type":"ContainerDied","Data":"6416ca5e28d5c38cd725e39b0c0fcd60d6b25ea3b0f2266ae3cb7b3026b57b6d"}
Oct 30 12:58:08.830468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.830440    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn" event={"ID":"5fb623f7-f001-4d88-b898-4e50b421b05b","Type":"ContainerDied","Data":"ca3fa8dc343c561abcb035d04f5e5f07bf243cffdfd18920da344e5c4825146a"}
Oct 30 12:58:08.830628 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.830475    4522 scope.go:122] "RemoveContainer" containerID="6416ca5e28d5c38cd725e39b0c0fcd60d6b25ea3b0f2266ae3cb7b3026b57b6d"
Oct 30 12:58:08.830681 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.830629    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"
Oct 30 12:58:08.858651 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.858612    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb623f7-f001-4d88-b898-4e50b421b05b-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:08.895464 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.893592    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb623f7-f001-4d88-b898-4e50b421b05b-kube-api-access-zqxj6" pod "5fb623f7-f001-4d88-b898-4e50b421b05b" (UID: "5fb623f7-f001-4d88-b898-4e50b421b05b"). InnerVolumeSpecName "kube-api-access-zqxj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:08.904931 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.904580    4522 scope.go:122] "RemoveContainer" containerID="6416ca5e28d5c38cd725e39b0c0fcd60d6b25ea3b0f2266ae3cb7b3026b57b6d"
Oct 30 12:58:08.905769 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:08.905659    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6416ca5e28d5c38cd725e39b0c0fcd60d6b25ea3b0f2266ae3cb7b3026b57b6d\": container with ID starting with 6416ca5e28d5c38cd725e39b0c0fcd60d6b25ea3b0f2266ae3cb7b3026b57b6d not found: ID does not exist" containerID="6416ca5e28d5c38cd725e39b0c0fcd60d6b25ea3b0f2266ae3cb7b3026b57b6d"
Oct 30 12:58:08.905769 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.905704    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6416ca5e28d5c38cd725e39b0c0fcd60d6b25ea3b0f2266ae3cb7b3026b57b6d"} err="failed to get container status \"6416ca5e28d5c38cd725e39b0c0fcd60d6b25ea3b0f2266ae3cb7b3026b57b6d\": rpc error: code = NotFound desc = could not find container \"6416ca5e28d5c38cd725e39b0c0fcd60d6b25ea3b0f2266ae3cb7b3026b57b6d\": container with ID starting with 6416ca5e28d5c38cd725e39b0c0fcd60d6b25ea3b0f2266ae3cb7b3026b57b6d not found: ID does not exist"
Oct 30 12:58:08.959829 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:08.959797    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqxj6\" (UniqueName: \"kubernetes.io/projected/5fb623f7-f001-4d88-b898-4e50b421b05b-kube-api-access-zqxj6\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:09.185862 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:09.185816    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"]
Oct 30 12:58:09.191588 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:09.191554    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2bfb1b-57fc8f9f4c-vgbrn"]
Oct 30 12:58:09.654316 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:09.654273    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:09.872408 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:09.872347    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:10.334755 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:10.334720    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb623f7-f001-4d88-b898-4e50b421b05b" path="/var/lib/kubelet/pods/5fb623f7-f001-4d88-b898-4e50b421b05b/volumes"
Oct 30 12:58:10.504454 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:10.504415    4522 patch_prober.go:29] interesting pod/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:58:10.504622 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:10.504476    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:58:11.658377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:11.657721    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:13.394829 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.394795    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:13.853804 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.853776    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:58:13.929755 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.929666    4522 generic.go:356] "Generic (PLEG): container finished" podID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerID="31db14520b3ee36810b6ef7b63780e9a0919fddb3026a50a70a095bddce4ef26" exitCode=0
Oct 30 12:58:13.929755 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.929727    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"
Oct 30 12:58:13.929969 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.929752    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" event={"ID":"daea3195-b3c7-4500-9dfe-b753efa60e0e","Type":"ContainerDied","Data":"31db14520b3ee36810b6ef7b63780e9a0919fddb3026a50a70a095bddce4ef26"}
Oct 30 12:58:13.929969 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.929855    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b" event={"ID":"daea3195-b3c7-4500-9dfe-b753efa60e0e","Type":"ContainerDied","Data":"0eff509ccebf39580298e3c4ee9e472c24bf4aa9cfc25eab29e65ead951939fe"}
Oct 30 12:58:13.929969 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.929892    4522 scope.go:122] "RemoveContainer" containerID="31db14520b3ee36810b6ef7b63780e9a0919fddb3026a50a70a095bddce4ef26"
Oct 30 12:58:13.945107 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.945089    4522 scope.go:122] "RemoveContainer" containerID="2533e2550f8eb41c5a66db88615c8b11fca3bc7eae04269b6c821f1f56c22673"
Oct 30 12:58:13.954947 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.954913    4522 scope.go:122] "RemoveContainer" containerID="31db14520b3ee36810b6ef7b63780e9a0919fddb3026a50a70a095bddce4ef26"
Oct 30 12:58:13.955334 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:13.955306    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31db14520b3ee36810b6ef7b63780e9a0919fddb3026a50a70a095bddce4ef26\": container with ID starting with 31db14520b3ee36810b6ef7b63780e9a0919fddb3026a50a70a095bddce4ef26 not found: ID does not exist" containerID="31db14520b3ee36810b6ef7b63780e9a0919fddb3026a50a70a095bddce4ef26"
Oct 30 12:58:13.955415 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.955382    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31db14520b3ee36810b6ef7b63780e9a0919fddb3026a50a70a095bddce4ef26"} err="failed to get container status \"31db14520b3ee36810b6ef7b63780e9a0919fddb3026a50a70a095bddce4ef26\": rpc error: code = NotFound desc = could not find container \"31db14520b3ee36810b6ef7b63780e9a0919fddb3026a50a70a095bddce4ef26\": container with ID starting with 31db14520b3ee36810b6ef7b63780e9a0919fddb3026a50a70a095bddce4ef26 not found: ID does not exist"
Oct 30 12:58:13.955468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.955413    4522 scope.go:122] "RemoveContainer" containerID="2533e2550f8eb41c5a66db88615c8b11fca3bc7eae04269b6c821f1f56c22673"
Oct 30 12:58:13.955711 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:13.955692    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2533e2550f8eb41c5a66db88615c8b11fca3bc7eae04269b6c821f1f56c22673\": container with ID starting with 2533e2550f8eb41c5a66db88615c8b11fca3bc7eae04269b6c821f1f56c22673 not found: ID does not exist" containerID="2533e2550f8eb41c5a66db88615c8b11fca3bc7eae04269b6c821f1f56c22673"
Oct 30 12:58:13.955758 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.955720    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2533e2550f8eb41c5a66db88615c8b11fca3bc7eae04269b6c821f1f56c22673"} err="failed to get container status \"2533e2550f8eb41c5a66db88615c8b11fca3bc7eae04269b6c821f1f56c22673\": rpc error: code = NotFound desc = could not find container \"2533e2550f8eb41c5a66db88615c8b11fca3bc7eae04269b6c821f1f56c22673\": container with ID starting with 2533e2550f8eb41c5a66db88615c8b11fca3bc7eae04269b6c821f1f56c22673 not found: ID does not exist"
Oct 30 12:58:13.961705 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.961685    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-kube-api-access-24q9j\" (UniqueName: \"kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-kube-api-access-24q9j\") pod \"daea3195-b3c7-4500-9dfe-b753efa60e0e\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") "
Oct 30 12:58:13.961781 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.961725    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-envoy\") pod \"daea3195-b3c7-4500-9dfe-b753efa60e0e\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") "
Oct 30 12:58:13.961824 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.961789    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-token\" (UniqueName: \"kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-token\") pod \"daea3195-b3c7-4500-9dfe-b753efa60e0e\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") "
Oct 30 12:58:13.961866 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.961833    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daea3195-b3c7-4500-9dfe-b753efa60e0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daea3195-b3c7-4500-9dfe-b753efa60e0e-istiod-ca-cert\") pod \"daea3195-b3c7-4500-9dfe-b753efa60e0e\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") "
Oct 30 12:58:13.961911 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.961892    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-podinfo\") pod \"daea3195-b3c7-4500-9dfe-b753efa60e0e\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") "
Oct 30 12:58:13.961969 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.961938    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-certs\") pod \"daea3195-b3c7-4500-9dfe-b753efa60e0e\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") "
Oct 30 12:58:13.962025 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.961979    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-data\") pod \"daea3195-b3c7-4500-9dfe-b753efa60e0e\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") "
Oct 30 12:58:13.962097 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.962083    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-socket\") pod \"daea3195-b3c7-4500-9dfe-b753efa60e0e\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") "
Oct 30 12:58:13.962162 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.962104    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-credential-socket\") pod \"daea3195-b3c7-4500-9dfe-b753efa60e0e\" (UID: \"daea3195-b3c7-4500-9dfe-b753efa60e0e\") "
Oct 30 12:58:13.962254 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.962229    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daea3195-b3c7-4500-9dfe-b753efa60e0e-istiod-ca-cert" pod "daea3195-b3c7-4500-9dfe-b753efa60e0e" (UID: "daea3195-b3c7-4500-9dfe-b753efa60e0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:13.962394 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.962369    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-data" pod "daea3195-b3c7-4500-9dfe-b753efa60e0e" (UID: "daea3195-b3c7-4500-9dfe-b753efa60e0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:13.962467 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.962383    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-certs" pod "daea3195-b3c7-4500-9dfe-b753efa60e0e" (UID: "daea3195-b3c7-4500-9dfe-b753efa60e0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:13.962605 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.962581    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-socket" pod "daea3195-b3c7-4500-9dfe-b753efa60e0e" (UID: "daea3195-b3c7-4500-9dfe-b753efa60e0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:13.962605 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.962596    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-credential-socket" pod "daea3195-b3c7-4500-9dfe-b753efa60e0e" (UID: "daea3195-b3c7-4500-9dfe-b753efa60e0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:13.962866 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.962845    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.962944 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.962872    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.962944 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.962889    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.962944 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.962904    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daea3195-b3c7-4500-9dfe-b753efa60e0e-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.962944 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.962917    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.963827 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.963798    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-envoy" pod "daea3195-b3c7-4500-9dfe-b753efa60e0e" (UID: "daea3195-b3c7-4500-9dfe-b753efa60e0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:13.964003 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.963984    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-kube-api-access-24q9j" pod "daea3195-b3c7-4500-9dfe-b753efa60e0e" (UID: "daea3195-b3c7-4500-9dfe-b753efa60e0e"). InnerVolumeSpecName "kube-api-access-24q9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:13.964065 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.964040    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-token" pod "daea3195-b3c7-4500-9dfe-b753efa60e0e" (UID: "daea3195-b3c7-4500-9dfe-b753efa60e0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:13.964217 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:13.964198    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-podinfo" pod "daea3195-b3c7-4500-9dfe-b753efa60e0e" (UID: "daea3195-b3c7-4500-9dfe-b753efa60e0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:58:14.063888 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:14.063857    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:14.063888 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:14.063886    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24q9j\" (UniqueName: \"kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-kube-api-access-24q9j\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:14.064029 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:14.063896    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:14.064029 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:14.063906    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daea3195-b3c7-4500-9dfe-b753efa60e0e-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:14.276058 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:14.276028    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"]
Oct 30 12:58:14.287901 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:14.287874    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6a4499-openshift-default-8677876f85-2m44b"]
Oct 30 12:58:14.334924 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:14.334893    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" path="/var/lib/kubelet/pods/daea3195-b3c7-4500-9dfe-b753efa60e0e/volumes"
Oct 30 12:58:15.081470 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:15.081439    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:16.405817 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:16.405782    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:16.505002 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:16.504965    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:16.612219 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:16.611902    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:17.428493 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.428460    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:17.523215 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.523179    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:17.551060 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551031    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"]
Oct 30 12:58:17.551637 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551619    4522 cpu_manager.go:414] "containerMap: removing container" podUID="c4fd5c1b-b147-44be-b82a-a69fc5f2ee11" containerName="payload-pre-processing"
Oct 30 12:58:17.551637 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551638    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fd5c1b-b147-44be-b82a-a69fc5f2ee11" containerName="payload-pre-processing"
Oct 30 12:58:17.551737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551652    4522 cpu_manager.go:414] "containerMap: removing container" podUID="917fa195-0b4a-4cb8-8021-0573bcae153b" containerName="payload-pre-processing"
Oct 30 12:58:17.551737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551659    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917fa195-0b4a-4cb8-8021-0573bcae153b" containerName="payload-pre-processing"
Oct 30 12:58:17.551737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551666    4522 cpu_manager.go:414] "containerMap: removing container" podUID="c566749d-af52-4992-803c-334d8e0d08c6" containerName="maas-api"
Oct 30 12:58:17.551737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551671    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c566749d-af52-4992-803c-334d8e0d08c6" containerName="maas-api"
Oct 30 12:58:17.551737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551686    4522 cpu_manager.go:414] "containerMap: removing container" podUID="5fb623f7-f001-4d88-b898-4e50b421b05b" containerName="payload-pre-processing"
Oct 30 12:58:17.551737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551691    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb623f7-f001-4d88-b898-4e50b421b05b" containerName="payload-pre-processing"
Oct 30 12:58:17.551737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551707    4522 cpu_manager.go:414] "containerMap: removing container" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy"
Oct 30 12:58:17.551737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551711    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy"
Oct 30 12:58:17.551737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551718    4522 cpu_manager.go:414] "containerMap: removing container" podUID="8e969a4e-090b-4931-a158-8b45c1c29eb2" containerName="payload-pre-processing"
Oct 30 12:58:17.551737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551722    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e969a4e-090b-4931-a158-8b45c1c29eb2" containerName="payload-pre-processing"
Oct 30 12:58:17.551737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551728    4522 cpu_manager.go:414] "containerMap: removing container" podUID="53cab76d-b312-4afb-bc3d-c29ece805ce1" containerName="payload-processing"
Oct 30 12:58:17.551737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551732    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cab76d-b312-4afb-bc3d-c29ece805ce1" containerName="payload-processing"
Oct 30 12:58:17.551737 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551740    4522 cpu_manager.go:414] "containerMap: removing container" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551744    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551755    4522 cpu_manager.go:414] "containerMap: removing container" podUID="7237c611-57be-484a-b096-0e12164ab8eb" containerName="payload-processing"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551760    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7237c611-57be-484a-b096-0e12164ab8eb" containerName="payload-processing"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551768    4522 cpu_manager.go:414] "containerMap: removing container" podUID="08206131-6355-4479-bfaf-a9920a03bc91" containerName="payload-processing"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551772    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08206131-6355-4479-bfaf-a9920a03bc91" containerName="payload-processing"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551855    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551863    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e969a4e-090b-4931-a158-8b45c1c29eb2" containerName="payload-pre-processing"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551870    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="daea3195-b3c7-4500-9dfe-b753efa60e0e" containerName="istio-proxy"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551876    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fd5c1b-b147-44be-b82a-a69fc5f2ee11" containerName="payload-pre-processing"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551883    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c566749d-af52-4992-803c-334d8e0d08c6" containerName="maas-api"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551888    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="917fa195-0b4a-4cb8-8021-0573bcae153b" containerName="payload-pre-processing"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551895    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cab76d-b312-4afb-bc3d-c29ece805ce1" containerName="payload-processing"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551902    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="08206131-6355-4479-bfaf-a9920a03bc91" containerName="payload-processing"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551908    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb623f7-f001-4d88-b898-4e50b421b05b" containerName="payload-pre-processing"
Oct 30 12:58:17.552113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.551916    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="7237c611-57be-484a-b096-0e12164ab8eb" containerName="payload-processing"
Oct 30 12:58:17.555336 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.555319    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"
Oct 30 12:58:17.576676 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.576645    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"]
Oct 30 12:58:17.583765 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.583740    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7a04e002\""
Oct 30 12:58:17.583907 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.583805    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7a04e002-dockercfg-nkxtx\""
Oct 30 12:58:17.626170 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.626134    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk\" (UID: \"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1\") " pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"
Oct 30 12:58:17.626311 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.626283    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrphw\" (UniqueName: \"kubernetes.io/projected/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-kube-api-access-nrphw\") pod \"payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk\" (UID: \"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1\") " pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"
Oct 30 12:58:17.839056 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.838820    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk\" (UID: \"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1\") " pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"
Oct 30 12:58:17.852387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.840179    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrphw\" (UniqueName: \"kubernetes.io/projected/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-kube-api-access-nrphw\") pod \"payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk\" (UID: \"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1\") " pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"
Oct 30 12:58:17.852387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.841816    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:17.852387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.842871    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk\" (UID: \"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1\") " pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"
Oct 30 12:58:17.946381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:17.934369    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrphw\" (UniqueName: \"kubernetes.io/projected/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-kube-api-access-nrphw\") pod \"payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk\" (UID: \"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1\") " pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"
Oct 30 12:58:18.169596 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:18.169517    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"
Oct 30 12:58:18.470728 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:18.470699    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"]
Oct 30 12:58:18.471169 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:58:18.470802    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c6cfe2_0ab5_4033_b16c_ff5630f8f3e1.slice/crio-1d4c531f765acdbc8ea25edc2231f46ef56da9ce3bc70eb7e113f709665b24fb WatchSource:0}: Error finding container 1d4c531f765acdbc8ea25edc2231f46ef56da9ce3bc70eb7e113f709665b24fb: Status 404 returned error can't find the container with id 1d4c531f765acdbc8ea25edc2231f46ef56da9ce3bc70eb7e113f709665b24fb
Oct 30 12:58:19.090111 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:19.090075    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk" event={"ID":"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1","Type":"ContainerStarted","Data":"3ff16a62dfa840ab4e52180944e6812c45aa8535c8215422e01fcab033c6c347"}
Oct 30 12:58:19.090268 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:19.090136    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk" event={"ID":"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1","Type":"ContainerStarted","Data":"1d4c531f765acdbc8ea25edc2231f46ef56da9ce3bc70eb7e113f709665b24fb"}
Oct 30 12:58:19.090268 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:19.090175    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"
Oct 30 12:58:19.160679 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:19.160611    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk" podStartSLOduration=2.160589904 podStartE2EDuration="2.160589904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:19.157538997 +0000 UTC m=+1342.972865488" watchObservedRunningTime="2026-10-30 12:58:19.160589904 +0000 UTC m=+1342.975916395"
Oct 30 12:58:19.417532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:19.417451    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:20.985384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:20.985322    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"]
Oct 30 12:58:20.989510 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:20.989490    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.007982 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.007953    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"]
Oct 30 12:58:21.015826 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.015796    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7a04e002-serving-cert\""
Oct 30 12:58:21.030601 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.030564    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ad0ed5-2d34-4422-bcab-6b2ec770b307-openshift-service-ca\") pod \"maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.030789 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.030738    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfhpr\" (UniqueName: \"kubernetes.io/projected/16ad0ed5-2d34-4422-bcab-6b2ec770b307-kube-api-access-dfhpr\") pod \"maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.030891 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.030876    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-metrics-certs\") pod \"maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.030955 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.030907    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-maas-api-tls\") pod \"maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.066871 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.066835    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:21.132712 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.132678    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ad0ed5-2d34-4422-bcab-6b2ec770b307-openshift-service-ca\") pod \"maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.132916 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.132786    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfhpr\" (UniqueName: \"kubernetes.io/projected/16ad0ed5-2d34-4422-bcab-6b2ec770b307-kube-api-access-dfhpr\") pod \"maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.132985 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.132917    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-metrics-certs\") pod \"maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.132985 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.132947    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-maas-api-tls\") pod \"maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.133177 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:21.133150    4522 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-serving-cert: secret "maas-api-e2e-mt-7a04e002-serving-cert" not found
Oct 30 12:58:21.133250 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:21.133226    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-maas-api-tls podName:16ad0ed5-2d34-4422-bcab-6b2ec770b307 nodeName:}" failed. No retries permitted until 2026-10-30 12:58:21.633204986 +0000 UTC m=+1345.448531470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-maas-api-tls") pod "maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f" (UID: "16ad0ed5-2d34-4422-bcab-6b2ec770b307") : secret "maas-api-e2e-mt-7a04e002-serving-cert" not found
Oct 30 12:58:21.133528 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.133505    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ad0ed5-2d34-4422-bcab-6b2ec770b307-openshift-service-ca\") pod \"maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.135829 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.135806    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-metrics-certs\") pod \"maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.137559 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:21.137533    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:58:21.152608 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.152574    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfhpr\" (UniqueName: \"kubernetes.io/projected/16ad0ed5-2d34-4422-bcab-6b2ec770b307-kube-api-access-dfhpr\") pod \"maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.167618 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.167579    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:21.269195 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.269165    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:21.668026 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.667930    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-maas-api-tls\") pod \"maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.670402 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.670383    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-maas-api-tls\") pod \"maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:21.903874 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:21.903842    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:22.249289 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:22.241010    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"]
Oct 30 12:58:22.863860 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:22.863828    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:23.186263 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:23.186231    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:23.208455 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:23.208421    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f" event={"ID":"16ad0ed5-2d34-4422-bcab-6b2ec770b307","Type":"ContainerStarted","Data":"ff440e9e3628a63ebc4d359d51d8badfe51e072568b14584bdd588f3c93c582d"}
Oct 30 12:58:23.208626 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:23.208474    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f" event={"ID":"16ad0ed5-2d34-4422-bcab-6b2ec770b307","Type":"ContainerStarted","Data":"478f55d5625dd3fcab62146f3b49335a31dc34f81c04170569f38598cd76dcb4"}
Oct 30 12:58:23.208626 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:23.208545    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:23.246992 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:23.246946    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f" podStartSLOduration=2.8880788219999998 podStartE2EDuration="3.246924933s" totalImagesPullingTime="358.846111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:23.246227003 +0000 UTC m=+1347.061553491" watchObservedRunningTime="2026-10-30 12:58:23.246924933 +0000 UTC m=+1347.062251421"
Oct 30 12:58:23.655260 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:23.655216    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:23.768997 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:23.768965    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:24.764419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:24.764383    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:26.423830 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:26.423797    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:27.402188 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:27.402143    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:27.509768 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:27.509738    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:28.401511 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:28.401476    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:29.056823 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:29.056791    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:29.158051 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:29.158003    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:29.221941 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:29.221913    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:58:30.099912 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:30.099883    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"
Oct 30 12:58:30.122254 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:30.122226    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:31.484755 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:31.484723    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"]
Oct 30 12:58:31.485383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:31.484958    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k" podUID="ba60f1fb-d268-4846-b604-3cd81bd289e3" containerName="maas-api" containerID="cri-o://6bf591c9ac6a49ded37fa774270730eba82e28b6e86ef75fb8f3e10cb7ff43ee" gracePeriod=30
Oct 30 12:58:31.541930 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:31.541899    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:31.976418 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:31.972772    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:58:32.122432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.107877    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-maas-api-tls\") pod \"ba60f1fb-d268-4846-b604-3cd81bd289e3\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") "
Oct 30 12:58:32.122432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.108055    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba60f1fb-d268-4846-b604-3cd81bd289e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba60f1fb-d268-4846-b604-3cd81bd289e3-openshift-service-ca\") pod \"ba60f1fb-d268-4846-b604-3cd81bd289e3\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") "
Oct 30 12:58:32.122432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.108295    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba60f1fb-d268-4846-b604-3cd81bd289e3-kube-api-access-pflfw\" (UniqueName: \"kubernetes.io/projected/ba60f1fb-d268-4846-b604-3cd81bd289e3-kube-api-access-pflfw\") pod \"ba60f1fb-d268-4846-b604-3cd81bd289e3\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") "
Oct 30 12:58:32.122432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.108331    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-metrics-certs\") pod \"ba60f1fb-d268-4846-b604-3cd81bd289e3\" (UID: \"ba60f1fb-d268-4846-b604-3cd81bd289e3\") "
Oct 30 12:58:32.144419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.134204    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba60f1fb-d268-4846-b604-3cd81bd289e3-openshift-service-ca" pod "ba60f1fb-d268-4846-b604-3cd81bd289e3" (UID: "ba60f1fb-d268-4846-b604-3cd81bd289e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:32.168377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.159755    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:32.168377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.161829    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-metrics-certs" pod "ba60f1fb-d268-4846-b604-3cd81bd289e3" (UID: "ba60f1fb-d268-4846-b604-3cd81bd289e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:32.168377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.162029    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-maas-api-tls" pod "ba60f1fb-d268-4846-b604-3cd81bd289e3" (UID: "ba60f1fb-d268-4846-b604-3cd81bd289e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:32.173750 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.173622    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba60f1fb-d268-4846-b604-3cd81bd289e3-kube-api-access-pflfw" pod "ba60f1fb-d268-4846-b604-3cd81bd289e3" (UID: "ba60f1fb-d268-4846-b604-3cd81bd289e3"). InnerVolumeSpecName "kube-api-access-pflfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:32.210516 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.210474    4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba60f1fb-d268-4846-b604-3cd81bd289e3-openshift-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:32.210516 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.210515    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pflfw\" (UniqueName: \"kubernetes.io/projected/ba60f1fb-d268-4846-b604-3cd81bd289e3-kube-api-access-pflfw\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:32.210715 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.210532    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:32.210715 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.210545    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba60f1fb-d268-4846-b604-3cd81bd289e3-maas-api-tls\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:58:32.288213 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.288182    4522 generic.go:356] "Generic (PLEG): container finished" podID="ba60f1fb-d268-4846-b604-3cd81bd289e3" containerID="6bf591c9ac6a49ded37fa774270730eba82e28b6e86ef75fb8f3e10cb7ff43ee" exitCode=0
Oct 30 12:58:32.288527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.288228    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"
Oct 30 12:58:32.288527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.288267    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k" event={"ID":"ba60f1fb-d268-4846-b604-3cd81bd289e3","Type":"ContainerDied","Data":"6bf591c9ac6a49ded37fa774270730eba82e28b6e86ef75fb8f3e10cb7ff43ee"}
Oct 30 12:58:32.288527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.288332    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k" event={"ID":"ba60f1fb-d268-4846-b604-3cd81bd289e3","Type":"ContainerDied","Data":"93986ab34c55ddeb0c6cc558ed808dc9d9a27bb8a50c0a4de9ca0480eb19c111"}
Oct 30 12:58:32.288527 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.288469    4522 scope.go:122] "RemoveContainer" containerID="6bf591c9ac6a49ded37fa774270730eba82e28b6e86ef75fb8f3e10cb7ff43ee"
Oct 30 12:58:32.298829 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.298807    4522 scope.go:122] "RemoveContainer" containerID="6bf591c9ac6a49ded37fa774270730eba82e28b6e86ef75fb8f3e10cb7ff43ee"
Oct 30 12:58:32.299107 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:32.299089    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf591c9ac6a49ded37fa774270730eba82e28b6e86ef75fb8f3e10cb7ff43ee\": container with ID starting with 6bf591c9ac6a49ded37fa774270730eba82e28b6e86ef75fb8f3e10cb7ff43ee not found: ID does not exist" containerID="6bf591c9ac6a49ded37fa774270730eba82e28b6e86ef75fb8f3e10cb7ff43ee"
Oct 30 12:58:32.299163 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.299113    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf591c9ac6a49ded37fa774270730eba82e28b6e86ef75fb8f3e10cb7ff43ee"} err="failed to get container status \"6bf591c9ac6a49ded37fa774270730eba82e28b6e86ef75fb8f3e10cb7ff43ee\": rpc error: code = NotFound desc = could not find container \"6bf591c9ac6a49ded37fa774270730eba82e28b6e86ef75fb8f3e10cb7ff43ee\": container with ID starting with 6bf591c9ac6a49ded37fa774270730eba82e28b6e86ef75fb8f3e10cb7ff43ee not found: ID does not exist"
Oct 30 12:58:32.319229 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.319199    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:32.339750 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.339726    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"]
Oct 30 12:58:32.351374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.351339    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef1186-9654c9949-4rw4k"]
Oct 30 12:58:32.476801 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.474966    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:32.863344 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:32.863302    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:34.334807 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:34.334775    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba60f1fb-d268-4846-b604-3cd81bd289e3" path="/var/lib/kubelet/pods/ba60f1fb-d268-4846-b604-3cd81bd289e3/volumes"
Oct 30 12:58:34.576919 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:34.576889    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:34.919676 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:34.919647    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:34.987642 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:34.987609    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"]
Oct 30 12:58:34.988509 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:34.988487    4522 cpu_manager.go:414] "containerMap: removing container" podUID="ba60f1fb-d268-4846-b604-3cd81bd289e3" containerName="maas-api"
Oct 30 12:58:34.988573 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:34.988513    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba60f1fb-d268-4846-b604-3cd81bd289e3" containerName="maas-api"
Oct 30 12:58:34.988707 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:34.988694    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba60f1fb-d268-4846-b604-3cd81bd289e3" containerName="maas-api"
Oct 30 12:58:34.994456 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:34.994429    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.009104 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.009077    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f0a01c-openshift-default-dockercfg-7c58j\""
Oct 30 12:58:35.026626 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.026602    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"]
Oct 30 12:58:35.051487 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.051462    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-data\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.051625 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.051589    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-socket\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.051763 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.051628    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istiod-ca-cert\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.051763 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.051647    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-credential-socket\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.051763 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.051665    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qth92\" (UniqueName: \"kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-kube-api-access-qth92\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.052267 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.051805    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-certs\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.052267 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.051852    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-token\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.052267 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.051878    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-podinfo\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.052267 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.052002    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-envoy\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.053646 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.053623    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:35.155384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.154548    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-token\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.155384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.154611    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-podinfo\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.155384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.154666    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-envoy\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.155384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.154761    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-data\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.155384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.154873    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-socket\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.155384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.154918    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istiod-ca-cert\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.155384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.154954    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-credential-socket\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.155384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.154983    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qth92\" (UniqueName: \"kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-kube-api-access-qth92\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.155384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.155092    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-certs\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.155986 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.155578    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-certs\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.208432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.208160    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-token\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.212383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.208722    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-socket\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.212383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.211564    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-credential-socket\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.212383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.212210    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istiod-ca-cert\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.242392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.238748    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-data\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.274708 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.274665    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 12:58:35.276438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.276400    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-envoy\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.287279 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.287244    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qth92\" (UniqueName: \"kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-kube-api-access-qth92\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.288053 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.288027    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-podinfo\") pod \"e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") " pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.349703 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.349670    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:35.548158 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.545004    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:35.952971 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:58:35.952942    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c66945_0e5f_44d3_b5ec_228ddc468ffb.slice/crio-e767d2c1b16a4de09fe9b5ac22006d2fc62594bcb88b1036540be3b4af3d1cf1 WatchSource:0}: Error finding container e767d2c1b16a4de09fe9b5ac22006d2fc62594bcb88b1036540be3b4af3d1cf1: Status 404 returned error can't find the container with id e767d2c1b16a4de09fe9b5ac22006d2fc62594bcb88b1036540be3b4af3d1cf1
Oct 30 12:58:35.953533 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.953511    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"]
Oct 30 12:58:35.955485 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.955454    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:35.955555 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.955516    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:35.955555 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:35.955545    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:36.336820 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:36.336738    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" event={"ID":"54c66945-0e5f-44d3-b5ec-228ddc468ffb","Type":"ContainerStarted","Data":"fc9886805ac5845ffdd9e65fc024783e920d62af173d339af8e96bb9a8035e26"}
Oct 30 12:58:36.336820 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:36.336784    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" event={"ID":"54c66945-0e5f-44d3-b5ec-228ddc468ffb","Type":"ContainerStarted","Data":"e767d2c1b16a4de09fe9b5ac22006d2fc62594bcb88b1036540be3b4af3d1cf1"}
Oct 30 12:58:36.350840 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:36.350812    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:36.384391 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:36.384335    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" podStartSLOduration=2.384322399 podStartE2EDuration="2.384322399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:36.381049112 +0000 UTC m=+1360.196375612" watchObservedRunningTime="2026-10-30 12:58:36.384322399 +0000 UTC m=+1360.199648888"
Oct 30 12:58:36.909946 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:36.909909    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:37.362103 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:37.361458    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:38.420378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:38.416434    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:38.420378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:38.418485    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 12:58:38.452970 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:38.452931    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:38.714555 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:38.691254    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:38.868408 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:38.867889    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:38.920420 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:38.869895    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:39.003478 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:38.963884    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:39.092237 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:39.092199    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:39.150378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:39.135469    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:39.171218 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:39.171163    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:39.348907 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:39.348861    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:39.399517 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:39.399468    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:39.426374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:39.425432    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:58:39.426906 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:39.426786    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:58:39.499719 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:39.498437    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:39.674431 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:39.665449    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:39.992765 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:39.960799    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:58:40.007053 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:39.994570    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:58:40.007860 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.007834    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:58:40.228378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.212174    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:58:40.261318 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.261233    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:58:40.288489 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.288449    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:40.316167 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.316126    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:58:40.349594 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.349533    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:40.350481 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.350457    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:58:40.424532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.424490    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:40.606396 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.598995    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:58:40.624392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.608210    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:40.624392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.609118    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded"
Oct 30 12:58:40.774485 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.774437    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:58:40.997432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.996796    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:58:41.022375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:40.998533    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:58:41.044588 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.044511    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:41.075344 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.075312    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:58:41.173015 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.172979    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:58:41.173531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.173508    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:58:41.197377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.174022    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:58:41.197897 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.197870    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:58:41.349377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.328663    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:58:41.349377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.349099    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:41.511376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.505820    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:41.616736 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.616640    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded"
Oct 30 12:58:41.659622 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.659577    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:58:41.714423 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.711772    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:58:41.787485 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.787437    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:58:41.788421 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.788388    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:58:41.851378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:41.826180    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:42.108557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.108506    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:42.268378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.243160    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:42.268378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.243510    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:42.287725 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.287690    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:58:42.318888 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.318839    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:58:42.405468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.339429    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:42.405468 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.379514    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:42.591657 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.591611    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:42.868439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.699534    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:42.868439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.723964    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:58:42.868439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.840675    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:58:42.868439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.841068    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:42.920378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.889454    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:58:42.967381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:42.954421    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:58:43.236433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.167754    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:58:43.275242 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.274419    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:43.290375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.274460    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:58:43.446692 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.382471    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:58:43.541496 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.399453    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:43.541496 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.399491    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:58:43.556135 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.556101    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:43.557236 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.557197    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:43.565032 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.565000    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:43.728381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.720745    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:43.728381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.725316    4522 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:58:43.787134 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.787000    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:43.788707 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.788665    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:58:43.865677 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.865558    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:58:43.870209 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.868826    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:58:43.897572 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.897518    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:43.937091 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:43.936565    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:44.163348 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.163260    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"]
Oct 30 12:58:44.285377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.265515    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:44.286484 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.286446    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"
Oct 30 12:58:44.352381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.288112    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"]
Oct 30 12:58:44.383476 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.288542    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:44.393557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.288576    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:44.406377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.303249    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f0a01c\""
Oct 30 12:58:44.406601 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.315895    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f0a01c-dockercfg-rjpg5\""
Oct 30 12:58:44.406730 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.352583    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:44.411432 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.373119    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqhn7\" (UniqueName: \"kubernetes.io/projected/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-kube-api-access-wqhn7\") pod \"payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88\" (UID: \"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"
Oct 30 12:58:44.444453 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.416066    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88\" (UID: \"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"
Oct 30 12:58:44.522388 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.480687    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:58:44.550685 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.523218    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:58:44.550963 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.523255    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:58:44.557742 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.551444    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhn7\" (UniqueName: \"kubernetes.io/projected/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-kube-api-access-wqhn7\") pod \"payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88\" (UID: \"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"
Oct 30 12:58:44.557742 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.551605    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88\" (UID: \"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"
Oct 30 12:58:44.557742 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.552760    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88\" (UID: \"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"
Oct 30 12:58:44.790496 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.790439    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:44.791977 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.791936    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:44.826153 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.826106    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:44.893381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.871804    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:45.020378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.916562    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:45.020378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.976871    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:45.020378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.917454    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:45.020378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:44.999341    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhn7\" (UniqueName: \"kubernetes.io/projected/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-kube-api-access-wqhn7\") pod \"payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88\" (UID: \"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"
Oct 30 12:58:45.208760 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:45.208658    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"
Oct 30 12:58:45.402378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:45.397656    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:45.402378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:45.401859    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:45.410466 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:45.410423    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:45.454986 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:45.454952    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:58:45.560570 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:45.560527    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:58:45.587951 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:45.587909    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:45.588460 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:45.588431    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:58:45.769420 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:45.769376    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:45.793458 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:45.793419    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:46.002308 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.002247    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:58:46.124384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.002861    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:58:46.124384 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.023421    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 12:58:46.210433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.198462    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:46.210433 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.198531    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:46.469248 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.426281    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:58:46.469248 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.426542    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:46.469248 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.440826    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:46.618378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.598027    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:58:46.618378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.598274    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 12:58:46.643774 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.628417    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:46.711436 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.708459    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:46.845377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.833484    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:46.913676 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:46.913639    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"]
Oct 30 12:58:47.146525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.125445    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:47.146525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.125775    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:58:47.146525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.126013    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 12:58:47.146525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.136643    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88" event={"ID":"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5","Type":"ContainerStarted","Data":"d54bbe264f4da89825230b6ffec3debb67e7fa1dd48b0b146f77ca5889bacf48"}
Oct 30 12:58:47.368377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.364484    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:47.368377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.364546    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:47.830375 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.820513    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:47.862413 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.834455    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:47.898730 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.857656    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 12:58:47.913424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.907441    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:58:47.913424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.864599    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 12:58:47.913424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.864632    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 12:58:47.913424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.864654    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 12:58:47.913424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.864673    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 12:58:47.913424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.877619    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 12:58:47.913424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.877667    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 12:58:47.913424 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.877695    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 12:58:47.930009 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:47.921701    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:48.058787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.034303    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 12:58:48.058787 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.036008    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 12:58:48.128585 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.128488    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:48.207371 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.128830    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 12:58:48.289372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.274665    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 12:58:48.289372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.279827    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 12:58:48.289372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.282288    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 12:58:48.299660 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.299637    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 12:58:48.301335 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.301310    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 12:58:48.310958 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.310934    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 12:58:48.311156 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.311144    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 12:58:48.311444 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.311432    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 12:58:48.329653 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.326569    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 12:58:48.363372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.362978    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88" event={"ID":"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5","Type":"ContainerStarted","Data":"fd444614f40e78814fb28a9adba0d21ae2c28701f01080d76aa2092445176c75"}
Oct 30 12:58:48.363372 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.363050    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"
Oct 30 12:58:48.573896 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.573861    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:48.705907 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:48.705854    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88" podStartSLOduration=4.705840048 podStartE2EDuration="4.705840048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:48.702841512 +0000 UTC m=+1372.518168002" watchObservedRunningTime="2026-10-30 12:58:48.705840048 +0000 UTC m=+1372.521166545"
Oct 30 12:58:49.809736 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:49.809704    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:51.134632 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:51.134596    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:58:51.491533 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.491462    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"]
Oct 30 12:58:51.495671 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.495647    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:51.509472 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.509445    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-metrics-certs\") pod \"maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:51.509582 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.509512    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f58z\" (UniqueName: \"kubernetes.io/projected/44ef1e1c-9754-4027-a727-2fafe7aded64-kube-api-access-6f58z\") pod \"maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:51.509640 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.509625    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:51.509679 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.509658    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ef1e1c-9754-4027-a727-2fafe7aded64-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:51.529119 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.529089    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"]
Oct 30 12:58:51.529283 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.529212    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-dea6ee-serving-cert\""
Oct 30 12:58:51.562811 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.562779    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:51.611540 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.611511    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-metrics-certs\") pod \"maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:51.611735 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.611582    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f58z\" (UniqueName: \"kubernetes.io/projected/44ef1e1c-9754-4027-a727-2fafe7aded64-kube-api-access-6f58z\") pod \"maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:51.611735 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.611694    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:51.611735 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.611721    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ef1e1c-9754-4027-a727-2fafe7aded64-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:51.611901 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:51.611841    4522 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-serving-cert: secret "maas-api-e2e-ipp-a-dea6ee-serving-cert" not found
Oct 30 12:58:51.611952 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:58:51.611921    4522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-maas-api-tls podName:44ef1e1c-9754-4027-a727-2fafe7aded64 nodeName:}" failed. No retries permitted until 2026-10-30 12:58:52.111899287 +0000 UTC m=+1375.927225768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-maas-api-tls") pod "maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq" (UID: "44ef1e1c-9754-4027-a727-2fafe7aded64") : secret "maas-api-e2e-ipp-a-dea6ee-serving-cert" not found
Oct 30 12:58:51.612409 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.612388    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ef1e1c-9754-4027-a727-2fafe7aded64-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:51.613896 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.613875    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-metrics-certs\") pod \"maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:51.631305 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.631282    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f58z\" (UniqueName: \"kubernetes.io/projected/44ef1e1c-9754-4027-a727-2fafe7aded64-kube-api-access-6f58z\") pod \"maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:51.648910 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.648880    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:51.747409 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:51.747299    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:52.125148 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:52.125112    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:52.127492 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:52.127468    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:52.273327 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:52.273265    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:52.425983 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:52.425271    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:52.724601 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:52.724496    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:52.758108 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:58:52.758080    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ef1e1c_9754_4027_a727_2fafe7aded64.slice/crio-b311dfcac9f414820ff5b066a4edbceef90c7acb248a468d6874f69e260d67af WatchSource:0}: Error finding container b311dfcac9f414820ff5b066a4edbceef90c7acb248a468d6874f69e260d67af: Status 404 returned error can't find the container with id b311dfcac9f414820ff5b066a4edbceef90c7acb248a468d6874f69e260d67af
Oct 30 12:58:52.759095 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:52.759074    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"]
Oct 30 12:58:52.907009 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:52.906898    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:53.430152 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:53.430114    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq" event={"ID":"44ef1e1c-9754-4027-a727-2fafe7aded64","Type":"ContainerStarted","Data":"5eff633cc069a1b4a834a3f55cc61204d195ad9f19920cf9aab4f75109e39a03"}
Oct 30 12:58:53.430634 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:53.430174    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq" event={"ID":"44ef1e1c-9754-4027-a727-2fafe7aded64","Type":"ContainerStarted","Data":"b311dfcac9f414820ff5b066a4edbceef90c7acb248a468d6874f69e260d67af"}
Oct 30 12:58:53.430634 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:53.430203    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:53.475113 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:53.475023    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq" podStartSLOduration=2.019843695 podStartE2EDuration="2.475009725s" totalImagesPullingTime="455.16603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:53.473218769 +0000 UTC m=+1377.288545257" watchObservedRunningTime="2026-10-30 12:58:53.475009725 +0000 UTC m=+1377.290336211"
Oct 30 12:58:53.589095 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:53.589056    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:55.391707 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:55.391668    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:56.902021 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:56.901994    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:57.001455 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:57.001422    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:57.128575 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:57.128547    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:58.367284 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.367248    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"]
Oct 30 12:58:58.373061 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.373040    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.394261 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.394235    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"]
Oct 30 12:58:58.401649 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.401621    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2bfb1b-openshift-default-dockercfg-tgvhs\""
Oct 30 12:58:58.452972 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.452928    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d6p7\" (UniqueName: \"kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-kube-api-access-7d6p7\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.453155 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.453031    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-data\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.453155 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.453116    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8424038-b3b1-4839-9bc2-50ee8ef95419-istiod-ca-cert\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.453289 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.453234    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.453289 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.453254    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-token\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.453289 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.453273    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-podinfo\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.453452 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.453317    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-envoy\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.453452 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.453421    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-certs\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.453746 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.453718    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-credential-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.554668 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.554631    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-envoy\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.554839 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.554676    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-certs\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.554839 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.554702    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-credential-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.554839 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.554761    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d6p7\" (UniqueName: \"kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-kube-api-access-7d6p7\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.554956 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.554892    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-data\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.555030 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.555012    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8424038-b3b1-4839-9bc2-50ee8ef95419-istiod-ca-cert\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.555159 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.555138    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.555159 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.555141    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-certs\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.555387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.555161    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-credential-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.555387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.555170    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-token\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.555387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.555200    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-data\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.555387 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.555217    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-podinfo\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.555566 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.555397    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-socket\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.555837 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.555813    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8424038-b3b1-4839-9bc2-50ee8ef95419-istiod-ca-cert\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.557075 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.557053    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-envoy\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.557532 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.557515    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-podinfo\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.574311 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.574277    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-token\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.574446 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.574325    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d6p7\" (UniqueName: \"kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-kube-api-access-7d6p7\") pod \"e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") " pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:58.688064 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:58.687983    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:58:59.006303 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:59.006277    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:59.007150 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:58:59.007127    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8424038_b3b1_4839_9bc2_50ee8ef95419.slice/crio-15c26d8ba13bc85a4af0ab212250e5172fcd8f08a910b21f4b7f86d7cc528eeb WatchSource:0}: Error finding container 15c26d8ba13bc85a4af0ab212250e5172fcd8f08a910b21f4b7f86d7cc528eeb: Status 404 returned error can't find the container with id 15c26d8ba13bc85a4af0ab212250e5172fcd8f08a910b21f4b7f86d7cc528eeb
Oct 30 12:58:59.007512 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:59.007486    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"]
Oct 30 12:58:59.010405 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:59.010308    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:59.010504 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:59.010410    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:59.010504 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:59.010456    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:59.098412 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:59.098367    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:59.413595 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:59.413562    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"
Oct 30 12:58:59.465382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:59.464631    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 12:58:59.576647 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:59.559683    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc" event={"ID":"d8424038-b3b1-4839-9bc2-50ee8ef95419","Type":"ContainerStarted","Data":"9557a40c3d0ff700121421fef27f3027a8c7fb55149cd3c287a28a457b9bfff0"}
Oct 30 12:58:59.576647 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:59.559753    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc" event={"ID":"d8424038-b3b1-4839-9bc2-50ee8ef95419","Type":"ContainerStarted","Data":"15c26d8ba13bc85a4af0ab212250e5172fcd8f08a910b21f4b7f86d7cc528eeb"}
Oct 30 12:58:59.634217 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:59.634149    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc" podStartSLOduration=1.634128949 podStartE2EDuration="1.634128949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:59.63219654 +0000 UTC m=+1383.447523055" watchObservedRunningTime="2026-10-30 12:58:59.634128949 +0000 UTC m=+1383.449455439"
Oct 30 12:58:59.688304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:58:59.688216    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:59:00.411419 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:00.404877    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:00.504461 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:00.503330    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:00.697619 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:00.697343    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 12:59:00.697619 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:00.697429    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc" podUID="d8424038-b3b1-4839-9bc2-50ee8ef95419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 12:59:01.594589 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:01.594546    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:01.725876 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:01.716291    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 12:59:01.727418 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:01.727371    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc" podUID="d8424038-b3b1-4839-9bc2-50ee8ef95419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 12:59:02.688829 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:02.688785    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 12:59:02.689601 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:02.689565    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc" podUID="d8424038-b3b1-4839-9bc2-50ee8ef95419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 12:59:03.525914 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:03.525877    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:03.697376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:03.689806    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 12:59:03.697376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:03.689894    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc" podUID="d8424038-b3b1-4839-9bc2-50ee8ef95419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 12:59:03.708513 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:03.708463    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:03.819506 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:03.819410    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:04.689752 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:04.689712    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 12:59:04.690066 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:04.690034    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc" podUID="d8424038-b3b1-4839-9bc2-50ee8ef95419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 12:59:05.723379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:05.721806    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 12:59:05.723379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:05.721909    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc" podUID="d8424038-b3b1-4839-9bc2-50ee8ef95419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 12:59:06.592566 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:06.592525    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:06.693225 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:06.693193    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:06.693550 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:06.693536    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:59:06.694008 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:06.693991    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:59:06.694773 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:06.694754    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 12:59:07.322037 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:07.322000    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:08.031382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:08.031322    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"]
Oct 30 12:59:08.161865 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:08.161828    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:08.296900 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:08.296792    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:09.225662 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:09.225628    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:10.913846 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:10.913809    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:12.600920 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:12.600821    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:13.031857 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.031808    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk" podUID="79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1" containerName="payload-processing" containerID="cri-o://3ff16a62dfa840ab4e52180944e6812c45aa8535c8215422e01fcab033c6c347" gracePeriod=25
Oct 30 12:59:13.276668 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.276613    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:13.319137 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.319112    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"
Oct 30 12:59:13.359549 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.359506    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:13.444523 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.444451    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:13.535427 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.529942    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-kube-api-access-nrphw\" (UniqueName: \"kubernetes.io/projected/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-kube-api-access-nrphw\") pod \"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1\" (UID: \"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1\") "
Oct 30 12:59:13.535427 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.530167    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-plugins-config-volume\") pod \"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1\" (UID: \"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1\") "
Oct 30 12:59:13.539320 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.537997    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-plugins-config-volume" pod "79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1" (UID: "79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:13.539767 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.539734    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-kube-api-access-nrphw" pod "79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1" (UID: "79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1"). InnerVolumeSpecName "kube-api-access-nrphw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:13.644909 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.640763    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrphw\" (UniqueName: \"kubernetes.io/projected/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-kube-api-access-nrphw\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:59:13.645725 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.645699    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:59:13.900437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.894213    4522 generic.go:356] "Generic (PLEG): container finished" podID="79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1" containerID="3ff16a62dfa840ab4e52180944e6812c45aa8535c8215422e01fcab033c6c347" exitCode=0
Oct 30 12:59:13.900437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.898420    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk" event={"ID":"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1","Type":"ContainerDied","Data":"3ff16a62dfa840ab4e52180944e6812c45aa8535c8215422e01fcab033c6c347"}
Oct 30 12:59:13.900437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.898506    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk" event={"ID":"79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1","Type":"ContainerDied","Data":"1d4c531f765acdbc8ea25edc2231f46ef56da9ce3bc70eb7e113f709665b24fb"}
Oct 30 12:59:13.900437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.898563    4522 scope.go:122] "RemoveContainer" containerID="3ff16a62dfa840ab4e52180944e6812c45aa8535c8215422e01fcab033c6c347"
Oct 30 12:59:13.900437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.898787    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"
Oct 30 12:59:13.984376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.981558    4522 scope.go:122] "RemoveContainer" containerID="3ff16a62dfa840ab4e52180944e6812c45aa8535c8215422e01fcab033c6c347"
Oct 30 12:59:13.984376 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:59:13.982076    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff16a62dfa840ab4e52180944e6812c45aa8535c8215422e01fcab033c6c347\": container with ID starting with 3ff16a62dfa840ab4e52180944e6812c45aa8535c8215422e01fcab033c6c347 not found: ID does not exist" containerID="3ff16a62dfa840ab4e52180944e6812c45aa8535c8215422e01fcab033c6c347"
Oct 30 12:59:13.984376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:13.982135    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff16a62dfa840ab4e52180944e6812c45aa8535c8215422e01fcab033c6c347"} err="failed to get container status \"3ff16a62dfa840ab4e52180944e6812c45aa8535c8215422e01fcab033c6c347\": rpc error: code = NotFound desc = could not find container \"3ff16a62dfa840ab4e52180944e6812c45aa8535c8215422e01fcab033c6c347\": container with ID starting with 3ff16a62dfa840ab4e52180944e6812c45aa8535c8215422e01fcab033c6c347 not found: ID does not exist"
Oct 30 12:59:14.068205 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:14.068174    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"]
Oct 30 12:59:14.079740 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:14.079708    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a04e002-846dbb5b54-td7vk"]
Oct 30 12:59:14.335674 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:14.335637    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1" path="/var/lib/kubelet/pods/79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1/volumes"
Oct 30 12:59:14.640139 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:14.640068    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:16.376531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:16.375461    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:16.466545 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:16.466514    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:16.580409 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:16.574259    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:18.302176 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:18.302144    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:20.024301 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:20.024263    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:20.713244 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:20.713207    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:20.815049 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:20.815019    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:21.134546 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:59:21.134512    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:59:22.118369 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:22.118320    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:23.890033 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:23.889991    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:25.251945 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.251910    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"]
Oct 30 12:59:25.252427 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.252168    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f" podUID="16ad0ed5-2d34-4422-bcab-6b2ec770b307" containerName="maas-api" containerID="cri-o://ff440e9e3628a63ebc4d359d51d8badfe51e072568b14584bdd588f3c93c582d" gracePeriod=30
Oct 30 12:59:25.310014 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.309973    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:25.707376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.704769    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:59:25.781389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.779017    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-metrics-certs\") pod \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") "
Oct 30 12:59:25.781389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.779266    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-maas-api-tls\") pod \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") "
Oct 30 12:59:25.781389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.779299    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ad0ed5-2d34-4422-bcab-6b2ec770b307-kube-api-access-dfhpr\" (UniqueName: \"kubernetes.io/projected/16ad0ed5-2d34-4422-bcab-6b2ec770b307-kube-api-access-dfhpr\") pod \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") "
Oct 30 12:59:25.781389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.779432    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ad0ed5-2d34-4422-bcab-6b2ec770b307-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ad0ed5-2d34-4422-bcab-6b2ec770b307-openshift-service-ca\") pod \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\" (UID: \"16ad0ed5-2d34-4422-bcab-6b2ec770b307\") "
Oct 30 12:59:25.786571 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.781726    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ad0ed5-2d34-4422-bcab-6b2ec770b307-openshift-service-ca" pod "16ad0ed5-2d34-4422-bcab-6b2ec770b307" (UID: "16ad0ed5-2d34-4422-bcab-6b2ec770b307"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:25.848413 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.839666    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-maas-api-tls" pod "16ad0ed5-2d34-4422-bcab-6b2ec770b307" (UID: "16ad0ed5-2d34-4422-bcab-6b2ec770b307"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:59:25.848413 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.839820    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-metrics-certs" pod "16ad0ed5-2d34-4422-bcab-6b2ec770b307" (UID: "16ad0ed5-2d34-4422-bcab-6b2ec770b307"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:59:25.924374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.884054    4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ad0ed5-2d34-4422-bcab-6b2ec770b307-openshift-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:59:25.924374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.884094    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:59:25.924374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.884109    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ad0ed5-2d34-4422-bcab-6b2ec770b307-maas-api-tls\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:59:25.952530 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.952473    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ad0ed5-2d34-4422-bcab-6b2ec770b307-kube-api-access-dfhpr" pod "16ad0ed5-2d34-4422-bcab-6b2ec770b307" (UID: "16ad0ed5-2d34-4422-bcab-6b2ec770b307"). InnerVolumeSpecName "kube-api-access-dfhpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:25.990400 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:25.990307    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfhpr\" (UniqueName: \"kubernetes.io/projected/16ad0ed5-2d34-4422-bcab-6b2ec770b307-kube-api-access-dfhpr\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:59:26.107125 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.107037    4522 generic.go:356] "Generic (PLEG): container finished" podID="16ad0ed5-2d34-4422-bcab-6b2ec770b307" containerID="ff440e9e3628a63ebc4d359d51d8badfe51e072568b14584bdd588f3c93c582d" exitCode=0
Oct 30 12:59:26.107589 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.107567    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f" event={"ID":"16ad0ed5-2d34-4422-bcab-6b2ec770b307","Type":"ContainerDied","Data":"ff440e9e3628a63ebc4d359d51d8badfe51e072568b14584bdd588f3c93c582d"}
Oct 30 12:59:26.107740 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.107727    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f" event={"ID":"16ad0ed5-2d34-4422-bcab-6b2ec770b307","Type":"ContainerDied","Data":"478f55d5625dd3fcab62146f3b49335a31dc34f81c04170569f38598cd76dcb4"}
Oct 30 12:59:26.107843 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.107832    4522 scope.go:122] "RemoveContainer" containerID="ff440e9e3628a63ebc4d359d51d8badfe51e072568b14584bdd588f3c93c582d"
Oct 30 12:59:26.108079 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.108065    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"
Oct 30 12:59:26.141730 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.141702    4522 scope.go:122] "RemoveContainer" containerID="ff440e9e3628a63ebc4d359d51d8badfe51e072568b14584bdd588f3c93c582d"
Oct 30 12:59:26.142150 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:59:26.142123    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff440e9e3628a63ebc4d359d51d8badfe51e072568b14584bdd588f3c93c582d\": container with ID starting with ff440e9e3628a63ebc4d359d51d8badfe51e072568b14584bdd588f3c93c582d not found: ID does not exist" containerID="ff440e9e3628a63ebc4d359d51d8badfe51e072568b14584bdd588f3c93c582d"
Oct 30 12:59:26.142268 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.142180    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff440e9e3628a63ebc4d359d51d8badfe51e072568b14584bdd588f3c93c582d"} err="failed to get container status \"ff440e9e3628a63ebc4d359d51d8badfe51e072568b14584bdd588f3c93c582d\": rpc error: code = NotFound desc = could not find container \"ff440e9e3628a63ebc4d359d51d8badfe51e072568b14584bdd588f3c93c582d\": container with ID starting with ff440e9e3628a63ebc4d359d51d8badfe51e072568b14584bdd588f3c93c582d not found: ID does not exist"
Oct 30 12:59:26.162793 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.162763    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"]
Oct 30 12:59:26.173993 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.173967    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a04e002-57d96987f6-5ss2f"]
Oct 30 12:59:26.205632 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.205606    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:26.303946 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.303920    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:26.336857 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.336831    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ad0ed5-2d34-4422-bcab-6b2ec770b307" path="/var/lib/kubelet/pods/16ad0ed5-2d34-4422-bcab-6b2ec770b307/volumes"
Oct 30 12:59:26.392173 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.392108    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:26.821490 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:26.821458    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:27.875564 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:27.875534    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"]
Oct 30 12:59:27.876203 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:27.876170    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv" podUID="e7580498-3df4-4052-b222-4020e493c364" containerName="maas-api" containerID="cri-o://9893d71ac983219cb5c898c37862a4462616499f3379e519b95d11cb06fa9182" gracePeriod=30
Oct 30 12:59:27.937088 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:27.937054    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:28.283383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.269515    4522 generic.go:356] "Generic (PLEG): container finished" podID="e7580498-3df4-4052-b222-4020e493c364" containerID="9893d71ac983219cb5c898c37862a4462616499f3379e519b95d11cb06fa9182" exitCode=0
Oct 30 12:59:28.283383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.269694    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv" event={"ID":"e7580498-3df4-4052-b222-4020e493c364","Type":"ContainerDied","Data":"9893d71ac983219cb5c898c37862a4462616499f3379e519b95d11cb06fa9182"}
Oct 30 12:59:28.497378 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.487561    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:59:28.575494 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.557483    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:28.576582 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.576550    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-metrics-certs\") pod \"e7580498-3df4-4052-b222-4020e493c364\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") "
Oct 30 12:59:28.576835 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.576817    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7580498-3df4-4052-b222-4020e493c364-kube-api-access-n9n6j\" (UniqueName: \"kubernetes.io/projected/e7580498-3df4-4052-b222-4020e493c364-kube-api-access-n9n6j\") pod \"e7580498-3df4-4052-b222-4020e493c364\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") "
Oct 30 12:59:28.577160 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.577142    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-maas-api-tls\") pod \"e7580498-3df4-4052-b222-4020e493c364\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") "
Oct 30 12:59:28.577438 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.577418    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7580498-3df4-4052-b222-4020e493c364-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7580498-3df4-4052-b222-4020e493c364-openshift-service-ca\") pod \"e7580498-3df4-4052-b222-4020e493c364\" (UID: \"e7580498-3df4-4052-b222-4020e493c364\") "
Oct 30 12:59:28.578870 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.578846    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7580498-3df4-4052-b222-4020e493c364-openshift-service-ca" pod "e7580498-3df4-4052-b222-4020e493c364" (UID: "e7580498-3df4-4052-b222-4020e493c364"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:28.586427 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.585856    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7580498-3df4-4052-b222-4020e493c364-kube-api-access-n9n6j" pod "e7580498-3df4-4052-b222-4020e493c364" (UID: "e7580498-3df4-4052-b222-4020e493c364"). InnerVolumeSpecName "kube-api-access-n9n6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:28.605383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.603556    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-maas-api-tls" pod "e7580498-3df4-4052-b222-4020e493c364" (UID: "e7580498-3df4-4052-b222-4020e493c364"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:59:28.623382 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.622714    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-metrics-certs" pod "e7580498-3df4-4052-b222-4020e493c364" (UID: "e7580498-3df4-4052-b222-4020e493c364"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:59:28.667164 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.667125    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:28.680073 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.680042    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:59:28.680210 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.680089    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9n6j\" (UniqueName: \"kubernetes.io/projected/e7580498-3df4-4052-b222-4020e493c364-kube-api-access-n9n6j\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:59:28.680210 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.680100    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7580498-3df4-4052-b222-4020e493c364-maas-api-tls\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:59:28.680210 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.680112    4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7580498-3df4-4052-b222-4020e493c364-openshift-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:59:28.787392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:28.773019    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:29.172312 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:29.172244    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:29.276788 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:29.276767    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"
Oct 30 12:59:29.276913 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:29.276804    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv" event={"ID":"e7580498-3df4-4052-b222-4020e493c364","Type":"ContainerDied","Data":"8f1855e0ebae5b4fb73d539c61b94d92878f7f428816d241ea7166e5bfa958dc"}
Oct 30 12:59:29.276913 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:29.276891    4522 scope.go:122] "RemoveContainer" containerID="9893d71ac983219cb5c898c37862a4462616499f3379e519b95d11cb06fa9182"
Oct 30 12:59:29.327894 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:29.327867    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"]
Oct 30 12:59:29.341630 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:29.341598    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-70621c-65dddf75b6-j4rxv"]
Oct 30 12:59:29.619499 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:29.619474    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"]
Oct 30 12:59:29.748435 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:29.748402    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:29.869630 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:29.869529    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:29.981641 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:29.981602    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:30.397534 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:30.397493    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7580498-3df4-4052-b222-4020e493c364" path="/var/lib/kubelet/pods/e7580498-3df4-4052-b222-4020e493c364/volumes"
Oct 30 12:59:30.568072 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:30.568044    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:31.891297 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:31.891264    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:32.001497 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:32.001419    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:32.104212 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:32.094924    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:33.030668 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:33.030632    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:34.620439 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:34.620396    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88" podUID="d0c9f3dc-6991-49a7-ae2d-186341ec0bd5" containerName="payload-processing" containerID="cri-o://fd444614f40e78814fb28a9adba0d21ae2c28701f01080d76aa2092445176c75" gracePeriod=25
Oct 30 12:59:34.804754 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:34.804725    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:34.966247 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:34.965300    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"
Oct 30 12:59:35.000895 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.000825    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:35.061078 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.061032    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-plugins-config-volume\") pod \"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5\" (UID: \"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5\") "
Oct 30 12:59:35.061263 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.061108    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-kube-api-access-wqhn7\" (UniqueName: \"kubernetes.io/projected/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-kube-api-access-wqhn7\") pod \"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5\" (UID: \"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5\") "
Oct 30 12:59:35.061502 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.061472    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-plugins-config-volume" pod "d0c9f3dc-6991-49a7-ae2d-186341ec0bd5" (UID: "d0c9f3dc-6991-49a7-ae2d-186341ec0bd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:35.061701 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.061619    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:59:35.064172 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.064140    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-kube-api-access-wqhn7" pod "d0c9f3dc-6991-49a7-ae2d-186341ec0bd5" (UID: "d0c9f3dc-6991-49a7-ae2d-186341ec0bd5"). InnerVolumeSpecName "kube-api-access-wqhn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:35.099875 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.099777    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:35.181494 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.176249    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqhn7\" (UniqueName: \"kubernetes.io/projected/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5-kube-api-access-wqhn7\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 12:59:35.373799 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.373762    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:35.388886 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.388849    4522 generic.go:356] "Generic (PLEG): container finished" podID="d0c9f3dc-6991-49a7-ae2d-186341ec0bd5" containerID="fd444614f40e78814fb28a9adba0d21ae2c28701f01080d76aa2092445176c75" exitCode=0
Oct 30 12:59:35.389054 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.388904    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"
Oct 30 12:59:35.389054 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.388933    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88" event={"ID":"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5","Type":"ContainerDied","Data":"fd444614f40e78814fb28a9adba0d21ae2c28701f01080d76aa2092445176c75"}
Oct 30 12:59:35.389054 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.389001    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88" event={"ID":"d0c9f3dc-6991-49a7-ae2d-186341ec0bd5","Type":"ContainerDied","Data":"d54bbe264f4da89825230b6ffec3debb67e7fa1dd48b0b146f77ca5889bacf48"}
Oct 30 12:59:35.389054 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.389031    4522 scope.go:122] "RemoveContainer" containerID="fd444614f40e78814fb28a9adba0d21ae2c28701f01080d76aa2092445176c75"
Oct 30 12:59:35.399089 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.399074    4522 scope.go:122] "RemoveContainer" containerID="fd444614f40e78814fb28a9adba0d21ae2c28701f01080d76aa2092445176c75"
Oct 30 12:59:35.399330 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:59:35.399310    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd444614f40e78814fb28a9adba0d21ae2c28701f01080d76aa2092445176c75\": container with ID starting with fd444614f40e78814fb28a9adba0d21ae2c28701f01080d76aa2092445176c75 not found: ID does not exist" containerID="fd444614f40e78814fb28a9adba0d21ae2c28701f01080d76aa2092445176c75"
Oct 30 12:59:35.399395 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.399338    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd444614f40e78814fb28a9adba0d21ae2c28701f01080d76aa2092445176c75"} err="failed to get container status \"fd444614f40e78814fb28a9adba0d21ae2c28701f01080d76aa2092445176c75\": rpc error: code = NotFound desc = could not find container \"fd444614f40e78814fb28a9adba0d21ae2c28701f01080d76aa2092445176c75\": container with ID starting with fd444614f40e78814fb28a9adba0d21ae2c28701f01080d76aa2092445176c75 not found: ID does not exist"
Oct 30 12:59:35.485431 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.480439    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"]
Oct 30 12:59:35.513636 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:35.513603    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0a01c-74b677884d-spp88"]
Oct 30 12:59:36.335806 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:36.335774    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c9f3dc-6991-49a7-ae2d-186341ec0bd5" path="/var/lib/kubelet/pods/d0c9f3dc-6991-49a7-ae2d-186341ec0bd5/volumes"
Oct 30 12:59:36.629310 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:36.629227    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:36.967882 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:36.967801    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:37.363712 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.363679    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"]
Oct 30 12:59:37.364450 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.364428    4522 cpu_manager.go:414] "containerMap: removing container" podUID="e7580498-3df4-4052-b222-4020e493c364" containerName="maas-api"
Oct 30 12:59:37.364450 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.364450    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7580498-3df4-4052-b222-4020e493c364" containerName="maas-api"
Oct 30 12:59:37.364620 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.364463    4522 cpu_manager.go:414] "containerMap: removing container" podUID="d0c9f3dc-6991-49a7-ae2d-186341ec0bd5" containerName="payload-processing"
Oct 30 12:59:37.364620 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.364470    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c9f3dc-6991-49a7-ae2d-186341ec0bd5" containerName="payload-processing"
Oct 30 12:59:37.364620 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.364511    4522 cpu_manager.go:414] "containerMap: removing container" podUID="79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1" containerName="payload-processing"
Oct 30 12:59:37.364620 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.364519    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1" containerName="payload-processing"
Oct 30 12:59:37.364620 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.364544    4522 cpu_manager.go:414] "containerMap: removing container" podUID="16ad0ed5-2d34-4422-bcab-6b2ec770b307" containerName="maas-api"
Oct 30 12:59:37.364620 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.364550    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ad0ed5-2d34-4422-bcab-6b2ec770b307" containerName="maas-api"
Oct 30 12:59:37.364910 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.364695    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c6cfe2-0ab5-4033-b16c-ff5630f8f3e1" containerName="payload-processing"
Oct 30 12:59:37.364910 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.364712    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c9f3dc-6991-49a7-ae2d-186341ec0bd5" containerName="payload-processing"
Oct 30 12:59:37.364910 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.364722    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ad0ed5-2d34-4422-bcab-6b2ec770b307" containerName="maas-api"
Oct 30 12:59:37.364910 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.364731    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7580498-3df4-4052-b222-4020e493c364" containerName="maas-api"
Oct 30 12:59:37.370697 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.370674    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.400034 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.400007    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-026c2f-openshift-default-dockercfg-kcwm7\""
Oct 30 12:59:37.403467 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.403439    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"]
Oct 30 12:59:37.424377 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.424343    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:37.535376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.520811    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74749d96-9482-4a69-90b3-9be8c58faa09-istio-podinfo\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.562380 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.535647    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-data\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.562695 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.562662    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-credential-socket\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.562896 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.562879    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-socket\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.563151 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.563127    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-certs\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.563318 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.563298    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-envoy\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.563523 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.558893    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:37.563745 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.563573    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-istio-token\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.563931 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.563913    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v988m\" (UniqueName: \"kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-kube-api-access-v988m\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.564159 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.564114    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74749d96-9482-4a69-90b3-9be8c58faa09-istiod-ca-cert\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.671426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.669567    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74749d96-9482-4a69-90b3-9be8c58faa09-istio-podinfo\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.671426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.669626    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-data\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.671426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.669685    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-credential-socket\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.671426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.669739    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-socket\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.671426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.669790    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-certs\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.671426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.669820    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-envoy\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.671426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.669872    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-istio-token\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.671426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.669917    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v988m\" (UniqueName: \"kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-kube-api-access-v988m\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.671426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.670054    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74749d96-9482-4a69-90b3-9be8c58faa09-istiod-ca-cert\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.671426 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.670992    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74749d96-9482-4a69-90b3-9be8c58faa09-istiod-ca-cert\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.682200 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.682159    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-certs\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.682920 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.682894    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-data\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.683949 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.683309    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-credential-socket\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.736392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.684648    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-socket\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.736392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.699040    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74749d96-9482-4a69-90b3-9be8c58faa09-istio-podinfo\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.736392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.714345    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-istio-token\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.736392 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.717613    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-envoy\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.758557 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.756551    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v988m\" (UniqueName: \"kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-kube-api-access-v988m\") pod \"e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") " pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:37.985982 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:37.985894    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:38.432379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:38.423805    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"]
Oct 30 12:59:38.432379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:38.427662    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:38.432379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:38.427749    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:38.432379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:38.427798    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:38.548376 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:38.519278    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" event={"ID":"74749d96-9482-4a69-90b3-9be8c58faa09","Type":"ContainerStarted","Data":"6aeb39d18e18704017beed1ec964602974f80e2fc308d3dfb0b55e5aa61500a8"}
Oct 30 12:59:38.640381 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:38.640338    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:39.526606 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:39.526570    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" event={"ID":"74749d96-9482-4a69-90b3-9be8c58faa09","Type":"ContainerStarted","Data":"e465083f88d26f5a78107c7f001cd413bc0eb2fd7f039b92727fce371e7c8ad5"}
Oct 30 12:59:39.590032 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:39.589986    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" podStartSLOduration=2.589973892 podStartE2EDuration="2.589973892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:39.577762324 +0000 UTC m=+1423.393088813" watchObservedRunningTime="2026-10-30 12:59:39.589973892 +0000 UTC m=+1423.405300381"
Oct 30 12:59:39.986106 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:39.986020    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:39.991793 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:39.991765    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:40.419780 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:40.419750    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:40.556118 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:40.556082    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:40.568739 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:40.566647    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 12:59:41.932749 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:41.932715    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:42.034790 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:42.034748    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:42.767136 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:42.767093    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:42.860622 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:42.860585    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:42.957654 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:42.957600    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:44.221194 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.221153    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:44.314867 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.314821    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:44.389735 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.389702    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"]
Oct 30 12:59:44.394909 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.394887    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"
Oct 30 12:59:44.410608 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.410585    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"]
Oct 30 12:59:44.700715 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.700681    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-026c2f\""
Oct 30 12:59:44.732479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.730667    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-026c2f-dockercfg-7kqrg\""
Oct 30 12:59:44.732479 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.730942    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"]
Oct 30 12:59:44.766493 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.766436    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltf45\" (UniqueName: \"kubernetes.io/projected/6936032a-d78e-4d02-bb45-358368095ecf-kube-api-access-ltf45\") pod \"payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7\" (UID: \"6936032a-d78e-4d02-bb45-358368095ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"
Oct 30 12:59:44.767226 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.767195    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6936032a-d78e-4d02-bb45-358368095ecf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7\" (UID: \"6936032a-d78e-4d02-bb45-358368095ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"
Oct 30 12:59:44.842393 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.836285    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"]
Oct 30 12:59:44.842393 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.836425    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"
Oct 30 12:59:44.875653 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.875616    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6936032a-d78e-4d02-bb45-358368095ecf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7\" (UID: \"6936032a-d78e-4d02-bb45-358368095ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"
Oct 30 12:59:44.875830 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.875709    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4jnf\" (UniqueName: \"kubernetes.io/projected/b4a81a9a-e4ef-453a-8de4-96b9f403b134-kube-api-access-b4jnf\") pod \"payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn\" (UID: \"b4a81a9a-e4ef-453a-8de4-96b9f403b134\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"
Oct 30 12:59:44.875933 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.875908    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a81a9a-e4ef-453a-8de4-96b9f403b134-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn\" (UID: \"b4a81a9a-e4ef-453a-8de4-96b9f403b134\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"
Oct 30 12:59:44.876071 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.876050    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltf45\" (UniqueName: \"kubernetes.io/projected/6936032a-d78e-4d02-bb45-358368095ecf-kube-api-access-ltf45\") pod \"payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7\" (UID: \"6936032a-d78e-4d02-bb45-358368095ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"
Oct 30 12:59:44.876302 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.876282    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6936032a-d78e-4d02-bb45-358368095ecf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7\" (UID: \"6936032a-d78e-4d02-bb45-358368095ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"
Oct 30 12:59:44.895100 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.895074    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltf45\" (UniqueName: \"kubernetes.io/projected/6936032a-d78e-4d02-bb45-358368095ecf-kube-api-access-ltf45\") pod \"payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7\" (UID: \"6936032a-d78e-4d02-bb45-358368095ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"
Oct 30 12:59:44.966441 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.966339    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:44.977918 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.977895    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4jnf\" (UniqueName: \"kubernetes.io/projected/b4a81a9a-e4ef-453a-8de4-96b9f403b134-kube-api-access-b4jnf\") pod \"payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn\" (UID: \"b4a81a9a-e4ef-453a-8de4-96b9f403b134\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"
Oct 30 12:59:44.978044 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.977968    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a81a9a-e4ef-453a-8de4-96b9f403b134-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn\" (UID: \"b4a81a9a-e4ef-453a-8de4-96b9f403b134\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"
Oct 30 12:59:44.978623 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.978602    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a81a9a-e4ef-453a-8de4-96b9f403b134-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn\" (UID: \"b4a81a9a-e4ef-453a-8de4-96b9f403b134\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"
Oct 30 12:59:44.996886 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:44.996865    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4jnf\" (UniqueName: \"kubernetes.io/projected/b4a81a9a-e4ef-453a-8de4-96b9f403b134-kube-api-access-b4jnf\") pod \"payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn\" (UID: \"b4a81a9a-e4ef-453a-8de4-96b9f403b134\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"
Oct 30 12:59:45.012062 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:45.011824    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"
Oct 30 12:59:45.156389 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:45.153671    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"
Oct 30 12:59:45.469055 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:45.469028    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"]
Oct 30 12:59:45.470505 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:59:45.470475    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6936032a_d78e_4d02_bb45_358368095ecf.slice/crio-4e2ef238f4ad263c991fae912193fddf6fabb77ed82b2ce276c3fe386ed8f63c WatchSource:0}: Error finding container 4e2ef238f4ad263c991fae912193fddf6fabb77ed82b2ce276c3fe386ed8f63c: Status 404 returned error can't find the container with id 4e2ef238f4ad263c991fae912193fddf6fabb77ed82b2ce276c3fe386ed8f63c
Oct 30 12:59:45.483318 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:59:45.483289    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a81a9a_e4ef_453a_8de4_96b9f403b134.slice/crio-fbbbb5db0fd0b57a7b2162ed75a1f90924209670d6400c8f10ba6aa10422c3aa WatchSource:0}: Error finding container fbbbb5db0fd0b57a7b2162ed75a1f90924209670d6400c8f10ba6aa10422c3aa: Status 404 returned error can't find the container with id fbbbb5db0fd0b57a7b2162ed75a1f90924209670d6400c8f10ba6aa10422c3aa
Oct 30 12:59:45.485717 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:45.485693    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"]
Oct 30 12:59:45.847565 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:45.847528    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7" event={"ID":"6936032a-d78e-4d02-bb45-358368095ecf","Type":"ContainerStarted","Data":"8cfc99443dc24e6ba6cebe35e523de064eea80aac582b870f1ee9573db09cdff"}
Oct 30 12:59:45.847739 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:45.847595    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7" event={"ID":"6936032a-d78e-4d02-bb45-358368095ecf","Type":"ContainerStarted","Data":"4e2ef238f4ad263c991fae912193fddf6fabb77ed82b2ce276c3fe386ed8f63c"}
Oct 30 12:59:45.847739 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:45.847626    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"
Oct 30 12:59:45.849400 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:45.849344    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn" event={"ID":"b4a81a9a-e4ef-453a-8de4-96b9f403b134","Type":"ContainerStarted","Data":"0cb9cab5d374374ae8a39122926680ed8441d7219d8970713c3326f92a13252c"}
Oct 30 12:59:45.849510 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:45.849422    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"
Oct 30 12:59:45.849510 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:45.849439    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn" event={"ID":"b4a81a9a-e4ef-453a-8de4-96b9f403b134","Type":"ContainerStarted","Data":"fbbbb5db0fd0b57a7b2162ed75a1f90924209670d6400c8f10ba6aa10422c3aa"}
Oct 30 12:59:45.886430 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:45.886375    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7" podStartSLOduration=1.886348746 podStartE2EDuration="1.886348746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:45.884464027 +0000 UTC m=+1429.699790527" watchObservedRunningTime="2026-10-30 12:59:45.886348746 +0000 UTC m=+1429.701675234"
Oct 30 12:59:45.923503 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:45.923459    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn" podStartSLOduration=1.923445571 podStartE2EDuration="1.923445571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:45.921414047 +0000 UTC m=+1429.736740551" watchObservedRunningTime="2026-10-30 12:59:45.923445571 +0000 UTC m=+1429.738772061"
Oct 30 12:59:46.284630 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:46.284598    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:46.492695 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:46.492618    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:47.406341 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:47.406307    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:48.685904 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:48.685865    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:49.224219 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:49.224178    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:50.364872 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.364841    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:50.752704 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.752633    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"]
Oct 30 12:59:50.757641 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.757615    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.779512 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.779238    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-096cc3-openshift-default-dockercfg-5rpdm\""
Oct 30 12:59:50.794437 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.794406    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"]
Oct 30 12:59:50.811243 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.811206    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:50.815445 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.815415    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7695fab-0b10-4ffe-857e-32411056876c-istiod-ca-cert\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.815618 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.815599    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-certs\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.815675 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.815663    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-credential-socket\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.815786 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.815771    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-istio-token\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.815854 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.815841    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8g4g\" (UniqueName: \"kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-kube-api-access-p8g4g\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.815909 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.815874    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-data\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.815963 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.815942    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-envoy\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.816061 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.816046    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7695fab-0b10-4ffe-857e-32411056876c-istio-podinfo\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.816118 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.816085    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-socket\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.954383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.953772    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7695fab-0b10-4ffe-857e-32411056876c-istio-podinfo\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.954383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.953828    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-socket\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.954383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.953884    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7695fab-0b10-4ffe-857e-32411056876c-istiod-ca-cert\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.954383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.954062    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-certs\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.954383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.954124    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-credential-socket\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.954383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.954240    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-istio-token\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.954383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.954304    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g4g\" (UniqueName: \"kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-kube-api-access-p8g4g\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.954383 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.954329    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-data\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.955167 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.954415    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-envoy\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.963525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.955735    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-certs\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.963525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.959571    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-socket\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.963525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.960209    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-credential-socket\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.963525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.960939    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7695fab-0b10-4ffe-857e-32411056876c-istiod-ca-cert\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:50.963525 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.961536    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-data\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:51.003434 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:50.992306    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-istio-token\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:51.009374 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:51.005132    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7695fab-0b10-4ffe-857e-32411056876c-istio-podinfo\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:51.025102 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:51.023795    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8g4g\" (UniqueName: \"kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-kube-api-access-p8g4g\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:51.025102 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:51.024610    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:51.059061 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:51.050827    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-envoy\") pod \"e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") " pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:51.092379 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:51.091714    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:51.152536 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:51.146309    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:51.273435 ip-10-0-129-133 kubenswrapper[4522]: E1030 12:59:51.272161    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 12:59:51.585821 ip-10-0-129-133 kubenswrapper[4522]: W1030 12:59:51.585788    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7695fab_0b10_4ffe_857e_32411056876c.slice/crio-fa7d13347b64b9d96a7fe1d28e5c8cab28cb2389bd8cb951410608baf8fbde2f WatchSource:0}: Error finding container fa7d13347b64b9d96a7fe1d28e5c8cab28cb2389bd8cb951410608baf8fbde2f: Status 404 returned error can't find the container with id fa7d13347b64b9d96a7fe1d28e5c8cab28cb2389bd8cb951410608baf8fbde2f
Oct 30 12:59:51.586815 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:51.586793    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"]
Oct 30 12:59:51.588589 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:51.588548    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:51.588667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:51.588616    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:51.588667 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:51.588647    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:52.055329 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:52.055294    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql" event={"ID":"b7695fab-0b10-4ffe-857e-32411056876c","Type":"ContainerStarted","Data":"764a1ce65980d1d46d93c0561626b27d0275e415756884e64775d84280c35fba"}
Oct 30 12:59:52.055531 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:52.055346    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql" event={"ID":"b7695fab-0b10-4ffe-857e-32411056876c","Type":"ContainerStarted","Data":"fa7d13347b64b9d96a7fe1d28e5c8cab28cb2389bd8cb951410608baf8fbde2f"}
Oct 30 12:59:52.092751 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:52.092723    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:52.098553 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:52.098521    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:52.110669 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:52.110627    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql" podStartSLOduration=2.110610229 podStartE2EDuration="2.110610229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:52.108504341 +0000 UTC m=+1435.923830830" watchObservedRunningTime="2026-10-30 12:59:52.110610229 +0000 UTC m=+1435.925936717"
Oct 30 12:59:53.062535 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:53.062503    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:53.063517 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:53.063496    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 12:59:53.329811 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:53.329735    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:54.438914 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:54.438879    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:54.693256 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:54.693178    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:54.815304 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:54.815265    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:55.310608 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:55.310576    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:56.157752 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:56.157709    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:56.258894 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:56.258853    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:56.941202 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:56.941174    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"
Oct 30 12:59:56.941668 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:56.941649    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"
Oct 30 12:59:57.363312 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:57.363283    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:58.564846 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:58.564811    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:58.891851 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:58.891750    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:59.711145 ip-10-0-129-133 kubenswrapper[4522]: I1030 12:59:59.711107    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:00.071332 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.071227    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:00.148692 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.148657    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"]
Oct 30 13:00:00.154093 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.154075    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"
Oct 30 13:00:00.182728 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.182701    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"]
Oct 30 13:00:00.186866 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.186846    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"
Oct 30 13:00:00.188292 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.188268    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"]
Oct 30 13:00:00.193175 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.193156    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"
Oct 30 13:00:00.193683 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.193667    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"]
Oct 30 13:00:00.198814 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.198794    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"]
Oct 30 13:00:00.227753 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.227728    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"]
Oct 30 13:00:00.250550 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.250519    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645c0ee8-5706-4071-a097-22f266a84e55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7\" (UID: \"645c0ee8-5706-4071-a097-22f266a84e55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"
Oct 30 13:00:00.250697 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.250674    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cnrb\" (UniqueName: \"kubernetes.io/projected/645c0ee8-5706-4071-a097-22f266a84e55-kube-api-access-6cnrb\") pod \"maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7\" (UID: \"645c0ee8-5706-4071-a097-22f266a84e55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"
Oct 30 13:00:00.250806 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.250788    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jjbg\" (UniqueName: \"kubernetes.io/projected/5108bd61-b167-45ee-88f3-361559eab957-kube-api-access-7jjbg\") pod \"maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd\" (UID: \"5108bd61-b167-45ee-88f3-361559eab957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"
Oct 30 13:00:00.251026 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.251000    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5108bd61-b167-45ee-88f3-361559eab957-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd\" (UID: \"5108bd61-b167-45ee-88f3-361559eab957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"
Oct 30 13:00:00.252630 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.252609    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:00:00.352639 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.352563    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ca2fcf-24d8-409c-b345-c20dc74cbb34-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889420-vw4th\" (UID: \"87ca2fcf-24d8-409c-b345-c20dc74cbb34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"
Oct 30 13:00:00.352639 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.352632    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645c0ee8-5706-4071-a097-22f266a84e55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7\" (UID: \"645c0ee8-5706-4071-a097-22f266a84e55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"
Oct 30 13:00:00.352826 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.352713    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xrlm\" (UniqueName: \"kubernetes.io/projected/87ca2fcf-24d8-409c-b345-c20dc74cbb34-kube-api-access-4xrlm\") pod \"maas-api-key-cleanup-29889420-vw4th\" (UID: \"87ca2fcf-24d8-409c-b345-c20dc74cbb34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"
Oct 30 13:00:00.352826 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.352814    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cnrb\" (UniqueName: \"kubernetes.io/projected/645c0ee8-5706-4071-a097-22f266a84e55-kube-api-access-6cnrb\") pod \"maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7\" (UID: \"645c0ee8-5706-4071-a097-22f266a84e55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"
Oct 30 13:00:00.352913 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.352897    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jjbg\" (UniqueName: \"kubernetes.io/projected/5108bd61-b167-45ee-88f3-361559eab957-kube-api-access-7jjbg\") pod \"maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd\" (UID: \"5108bd61-b167-45ee-88f3-361559eab957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"
Oct 30 13:00:00.353060 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.353045    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5108bd61-b167-45ee-88f3-361559eab957-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd\" (UID: \"5108bd61-b167-45ee-88f3-361559eab957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"
Oct 30 13:00:00.353327 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.353304    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645c0ee8-5706-4071-a097-22f266a84e55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7\" (UID: \"645c0ee8-5706-4071-a097-22f266a84e55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"
Oct 30 13:00:00.353643 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.353625    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5108bd61-b167-45ee-88f3-361559eab957-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd\" (UID: \"5108bd61-b167-45ee-88f3-361559eab957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"
Oct 30 13:00:00.372261 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.372228    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cnrb\" (UniqueName: \"kubernetes.io/projected/645c0ee8-5706-4071-a097-22f266a84e55-kube-api-access-6cnrb\") pod \"maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7\" (UID: \"645c0ee8-5706-4071-a097-22f266a84e55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"
Oct 30 13:00:00.372400 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.372345    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jjbg\" (UniqueName: \"kubernetes.io/projected/5108bd61-b167-45ee-88f3-361559eab957-kube-api-access-7jjbg\") pod \"maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd\" (UID: \"5108bd61-b167-45ee-88f3-361559eab957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"
Oct 30 13:00:00.454520 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.454492    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ca2fcf-24d8-409c-b345-c20dc74cbb34-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889420-vw4th\" (UID: \"87ca2fcf-24d8-409c-b345-c20dc74cbb34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"
Oct 30 13:00:00.454672 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.454569    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrlm\" (UniqueName: \"kubernetes.io/projected/87ca2fcf-24d8-409c-b345-c20dc74cbb34-kube-api-access-4xrlm\") pod \"maas-api-key-cleanup-29889420-vw4th\" (UID: \"87ca2fcf-24d8-409c-b345-c20dc74cbb34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"
Oct 30 13:00:00.455260 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.455237    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ca2fcf-24d8-409c-b345-c20dc74cbb34-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889420-vw4th\" (UID: \"87ca2fcf-24d8-409c-b345-c20dc74cbb34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"
Oct 30 13:00:00.469787 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.469764    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"
Oct 30 13:00:00.481490 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.481469    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrlm\" (UniqueName: \"kubernetes.io/projected/87ca2fcf-24d8-409c-b345-c20dc74cbb34-kube-api-access-4xrlm\") pod \"maas-api-key-cleanup-29889420-vw4th\" (UID: \"87ca2fcf-24d8-409c-b345-c20dc74cbb34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"
Oct 30 13:00:00.508763 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.508730    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"
Oct 30 13:00:00.514746 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.514717    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"
Oct 30 13:00:00.819017 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.818918    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"]
Oct 30 13:00:00.819630 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:00:00.819599    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5108bd61_b167_45ee_88f3_361559eab957.slice/crio-0bb616b24e1c66d28e434b9d494c0108374c7ff39d53959b36aa93f253494242 WatchSource:0}: Error finding container 0bb616b24e1c66d28e434b9d494c0108374c7ff39d53959b36aa93f253494242: Status 404 returned error can't find the container with id 0bb616b24e1c66d28e434b9d494c0108374c7ff39d53959b36aa93f253494242
Oct 30 13:00:00.953679 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.953651    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"]
Oct 30 13:00:00.953679 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:00:00.953663    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645c0ee8_5706_4071_a097_22f266a84e55.slice/crio-7aa803586ea070df4e4012ef907d19c0215efa970bb8ea5f72812184c8d8e35f WatchSource:0}: Error finding container 7aa803586ea070df4e4012ef907d19c0215efa970bb8ea5f72812184c8d8e35f: Status 404 returned error can't find the container with id 7aa803586ea070df4e4012ef907d19c0215efa970bb8ea5f72812184c8d8e35f
Oct 30 13:00:00.988553 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:00:00.988523    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ca2fcf_24d8_409c_b345_c20dc74cbb34.slice/crio-e12d06bc938a1ae791188c6ad85f07cf0312027fb8f2aa2f2500fefdc079d4b8 WatchSource:0}: Error finding container e12d06bc938a1ae791188c6ad85f07cf0312027fb8f2aa2f2500fefdc079d4b8: Status 404 returned error can't find the container with id e12d06bc938a1ae791188c6ad85f07cf0312027fb8f2aa2f2500fefdc079d4b8
Oct 30 13:00:00.989745 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:00.989721    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"]
Oct 30 13:00:01.261550 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:01.261516    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd" event={"ID":"5108bd61-b167-45ee-88f3-361559eab957","Type":"ContainerStarted","Data":"147e24c3cd1834b9801871d8f1f7a10e2e6f548e385ea4892083f7aff07055e7"}
Oct 30 13:00:01.261707 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:01.261573    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd" event={"ID":"5108bd61-b167-45ee-88f3-361559eab957","Type":"ContainerStarted","Data":"0bb616b24e1c66d28e434b9d494c0108374c7ff39d53959b36aa93f253494242"}
Oct 30 13:00:01.262724 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:01.262698    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th" event={"ID":"87ca2fcf-24d8-409c-b345-c20dc74cbb34","Type":"ContainerStarted","Data":"e12d06bc938a1ae791188c6ad85f07cf0312027fb8f2aa2f2500fefdc079d4b8"}
Oct 30 13:00:01.263837 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:01.263820    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7" event={"ID":"645c0ee8-5706-4071-a097-22f266a84e55","Type":"ContainerStarted","Data":"7aa803586ea070df4e4012ef907d19c0215efa970bb8ea5f72812184c8d8e35f"}
Oct 30 13:00:01.303644 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:01.303599    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd" podStartSLOduration=1.303588131 podStartE2EDuration="1.303588131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:01.300821627 +0000 UTC m=+1445.116148117" watchObservedRunningTime="2026-10-30 13:00:01.303588131 +0000 UTC m=+1445.118914619"
Oct 30 13:00:01.465075 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:01.464997    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:02.271136 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:02.271103    4522 generic.go:356] "Generic (PLEG): container finished" podID="5108bd61-b167-45ee-88f3-361559eab957" containerID="147e24c3cd1834b9801871d8f1f7a10e2e6f548e385ea4892083f7aff07055e7" exitCode=60
Oct 30 13:00:02.271583 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:02.271191    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd" event={"ID":"5108bd61-b167-45ee-88f3-361559eab957","Type":"ContainerDied","Data":"147e24c3cd1834b9801871d8f1f7a10e2e6f548e385ea4892083f7aff07055e7"}
Oct 30 13:00:02.271583 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:02.271489    4522 scope.go:122] "RemoveContainer" containerID="147e24c3cd1834b9801871d8f1f7a10e2e6f548e385ea4892083f7aff07055e7"
Oct 30 13:00:02.272859 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:02.272839    4522 generic.go:356] "Generic (PLEG): container finished" podID="87ca2fcf-24d8-409c-b345-c20dc74cbb34" containerID="d3c63c2ff0d9641fd9c94182b8260af737063ac33cbdd452f2d1ec4c86a5e1be" exitCode=60
Oct 30 13:00:02.272959 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:02.272871    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th" event={"ID":"87ca2fcf-24d8-409c-b345-c20dc74cbb34","Type":"ContainerDied","Data":"d3c63c2ff0d9641fd9c94182b8260af737063ac33cbdd452f2d1ec4c86a5e1be"}
Oct 30 13:00:02.273163 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:02.273145    4522 scope.go:122] "RemoveContainer" containerID="d3c63c2ff0d9641fd9c94182b8260af737063ac33cbdd452f2d1ec4c86a5e1be"
Oct 30 13:00:02.275428 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:02.275405    4522 generic.go:356] "Generic (PLEG): container finished" podID="645c0ee8-5706-4071-a097-22f266a84e55" containerID="a0acdb5d4e3b72073d7b61ddcb980defe0f72576da0fac2feaa2f1e9b6540dfe" exitCode=60
Oct 30 13:00:02.275523 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:02.275456    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7" event={"ID":"645c0ee8-5706-4071-a097-22f266a84e55","Type":"ContainerDied","Data":"a0acdb5d4e3b72073d7b61ddcb980defe0f72576da0fac2feaa2f1e9b6540dfe"}
Oct 30 13:00:02.275710 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:02.275694    4522 scope.go:122] "RemoveContainer" containerID="a0acdb5d4e3b72073d7b61ddcb980defe0f72576da0fac2feaa2f1e9b6540dfe"
Oct 30 13:00:03.283256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:03.283221    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd" event={"ID":"5108bd61-b167-45ee-88f3-361559eab957","Type":"ContainerStarted","Data":"4511bf3b4c2a1a8295f1752420449b0b3d05fa89758a95294d2c79e6bcb74496"}
Oct 30 13:00:03.285266 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:03.285240    4522 generic.go:356] "Generic (PLEG): container finished" podID="87ca2fcf-24d8-409c-b345-c20dc74cbb34" containerID="84647a694baa373d405cf934a4a1f0ff0158a35fe617d1850d2c9a0f242f4408" exitCode=60
Oct 30 13:00:03.285410 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:03.285311    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th" event={"ID":"87ca2fcf-24d8-409c-b345-c20dc74cbb34","Type":"ContainerDied","Data":"84647a694baa373d405cf934a4a1f0ff0158a35fe617d1850d2c9a0f242f4408"}
Oct 30 13:00:03.285410 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:03.285388    4522 scope.go:122] "RemoveContainer" containerID="d3c63c2ff0d9641fd9c94182b8260af737063ac33cbdd452f2d1ec4c86a5e1be"
Oct 30 13:00:03.285604 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:03.285584    4522 scope.go:122] "RemoveContainer" containerID="84647a694baa373d405cf934a4a1f0ff0158a35fe617d1850d2c9a0f242f4408"
Oct 30 13:00:03.285920 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:03.285897    4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889420-vw4th_odh-ai-gateway-infra(87ca2fcf-24d8-409c-b345-c20dc74cbb34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th" podUID="87ca2fcf-24d8-409c-b345-c20dc74cbb34"
Oct 30 13:00:03.287473 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:03.287448    4522 generic.go:356] "Generic (PLEG): container finished" podID="645c0ee8-5706-4071-a097-22f266a84e55" containerID="0f8569816af6789dae6b8809a0eb3327df5f36bf24c332b92013d73df22f4369" exitCode=60
Oct 30 13:00:03.287566 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:03.287534    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7" event={"ID":"645c0ee8-5706-4071-a097-22f266a84e55","Type":"ContainerDied","Data":"0f8569816af6789dae6b8809a0eb3327df5f36bf24c332b92013d73df22f4369"}
Oct 30 13:00:03.287726 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:03.287711    4522 scope.go:122] "RemoveContainer" containerID="0f8569816af6789dae6b8809a0eb3327df5f36bf24c332b92013d73df22f4369"
Oct 30 13:00:03.287978 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:03.287960    4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7_odh-ai-gateway-infra(645c0ee8-5706-4071-a097-22f266a84e55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7" podUID="645c0ee8-5706-4071-a097-22f266a84e55"
Oct 30 13:00:03.297877 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:03.297858    4522 scope.go:122] "RemoveContainer" containerID="a0acdb5d4e3b72073d7b61ddcb980defe0f72576da0fac2feaa2f1e9b6540dfe"
Oct 30 13:00:03.958791 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:03.958760    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:04.099409 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:04.097965    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:04.189301 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:04.189266    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:04.295473 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:04.295441    4522 generic.go:356] "Generic (PLEG): container finished" podID="5108bd61-b167-45ee-88f3-361559eab957" containerID="4511bf3b4c2a1a8295f1752420449b0b3d05fa89758a95294d2c79e6bcb74496" exitCode=60
Oct 30 13:00:04.295934 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:04.295522    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd" event={"ID":"5108bd61-b167-45ee-88f3-361559eab957","Type":"ContainerDied","Data":"4511bf3b4c2a1a8295f1752420449b0b3d05fa89758a95294d2c79e6bcb74496"}
Oct 30 13:00:04.295934 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:04.295586    4522 scope.go:122] "RemoveContainer" containerID="147e24c3cd1834b9801871d8f1f7a10e2e6f548e385ea4892083f7aff07055e7"
Oct 30 13:00:04.295934 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:04.295757    4522 scope.go:122] "RemoveContainer" containerID="4511bf3b4c2a1a8295f1752420449b0b3d05fa89758a95294d2c79e6bcb74496"
Oct 30 13:00:04.296096 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:04.296071    4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd_odh-ai-gateway-infra(5108bd61-b167-45ee-88f3-361559eab957)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd" podUID="5108bd61-b167-45ee-88f3-361559eab957"
Oct 30 13:00:04.297518 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:04.297501    4522 scope.go:122] "RemoveContainer" containerID="84647a694baa373d405cf934a4a1f0ff0158a35fe617d1850d2c9a0f242f4408"
Oct 30 13:00:04.297807 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:04.297787    4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889420-vw4th_odh-ai-gateway-infra(87ca2fcf-24d8-409c-b345-c20dc74cbb34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th" podUID="87ca2fcf-24d8-409c-b345-c20dc74cbb34"
Oct 30 13:00:04.298901 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:04.298880    4522 scope.go:122] "RemoveContainer" containerID="0f8569816af6789dae6b8809a0eb3327df5f36bf24c332b92013d73df22f4369"
Oct 30 13:00:04.299131 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:04.299114    4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7_odh-ai-gateway-infra(645c0ee8-5706-4071-a097-22f266a84e55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7" podUID="645c0ee8-5706-4071-a097-22f266a84e55"
Oct 30 13:00:05.360137 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:05.360095    4522 scope.go:122] "RemoveContainer" containerID="4511bf3b4c2a1a8295f1752420449b0b3d05fa89758a95294d2c79e6bcb74496"
Oct 30 13:00:05.361301 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:05.361268    4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd_odh-ai-gateway-infra(5108bd61-b167-45ee-88f3-361559eab957)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd" podUID="5108bd61-b167-45ee-88f3-361559eab957"
Oct 30 13:00:06.019290 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:06.019252    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:07.268200 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:07.268170    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 13:00:08.004050 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:08.004015    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:08.761379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:08.757854    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:09.904184 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:09.904086    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:11.047718 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:11.047680    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:11.793831 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:11.793776    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:11.909382 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:11.908781    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.053446 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.052430    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.125510 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.125454    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.249484 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.249437    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.311548 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.311449    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.347230 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.347183    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.365389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.360400    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.392533 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.392484    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.537516 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.537464    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.801490 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.801443    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.810651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.810621    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:00:12.849604 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.815216    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.937680 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.937630    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:12.974380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.940236    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:00:12.995742 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:12.977014    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:00:13.074863 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.074433    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:13.090375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.075822    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 13:00:13.136822 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.135251    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:13.136822 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.135721    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:00:13.265979 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.265931    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:13.290376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.286659    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:00:13.351436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.321244    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:13.351436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.321791    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:00:13.378379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.366168    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:13.395086 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.379466    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:13.395086 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.381145    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:00:13.395086 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.381239    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:00:13.401378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.395815    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:13.401378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.396225    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:00:13.543345 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.543305    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:00:13.564487 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.564073    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:00:13.807107 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.807056    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:13.955561 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:13.955503    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:14.049513 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.049460    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:00:14.089433 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.050394    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:00:14.136629 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.136583    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:14.138027 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.137994    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:00:14.249531 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.249478    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:14.287477 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.287433    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:00:14.343436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.338511    4522 scope.go:122] "RemoveContainer" containerID="4511bf3b4c2a1a8295f1752420449b0b3d05fa89758a95294d2c79e6bcb74496"
Oct 30 13:00:14.365387 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.355007    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:14.389257 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.369662    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:14.465660 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.408270    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:14.465660 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.409013    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:00:14.593390 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.575458    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:00:14.723768 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.723723    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:14.832117 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.832050    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:14.832743 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.832698    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:00:14.985497 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.985451    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:14.985816 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:14.985791    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:00:15.736439 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.710480    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:15.736439 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.710768    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:00:15.769922 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.769880    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:15.804383 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.797707    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded"
Oct 30 13:00:15.822399 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.822337    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:00:15.822997 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.822966    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:00:15.824435 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.824409    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:00:16.131777 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.891309    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:16.132743 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.891386    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:00:16.133220 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.891415    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:00:16.133489 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.892242    4522 scope.go:122] "RemoveContainer" containerID="84647a694baa373d405cf934a4a1f0ff0158a35fe617d1850d2c9a0f242f4408"
Oct 30 13:00:16.134276 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:16.134254    4522 scope.go:122] "RemoveContainer" containerID="0f8569816af6789dae6b8809a0eb3327df5f36bf24c332b92013d73df22f4369"
Oct 30 13:00:16.135063 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.924437    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:16.135254 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.067338    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:16.135466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:15.902607    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:00:16.308376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:16.290418    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:16.311887 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:16.311829    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:16.312149 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:16.312119    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2" podUID="c3a4b170-e0c7-4b26-8817-aa969962c106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:16.773376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:16.711484    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:16.773376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:16.711903    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:00:16.841445 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:16.829229    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:00:17.142903 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:16.830497    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:00:17.156579 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:16.835456    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:17.230330 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:16.852792    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:17.230650 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.141541    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:00:17.231055 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.141581    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:00:17.231691 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.141608    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:00:17.232030 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.142437    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:00:17.232345 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.138698    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:00:17.236950 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.146408    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.241391 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:16.987454    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:17.241567 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.241539    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.245448 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.245418    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.314419 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.313803    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.419485 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.402013    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.428508 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.428469    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:17.436383 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.432433    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.630805 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:17.630769    4522 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.099s"
Oct 30 13:00:17.662559 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.662532    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"]
Oct 30 13:00:17.668505 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.639922    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:17.682952 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.682864    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"]
Oct 30 13:00:17.683302 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.683285    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"]
Oct 30 13:00:17.683655 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.683228    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.688952 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.688923    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6" podUID="b0519800-4332-4a1c-94a8-205367079050" containerName="istio-proxy" containerID="cri-o://2b2a584aa0dba1b26666e4275189b70c39f088c4d05c2ac8e5e3e130ca4f915a" gracePeriod=30
Oct 30 13:00:17.700542 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.698881    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-podinfo\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.700542 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.698923    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-data\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.700542 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.698972    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-socket\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.700542 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.699015    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-envoy\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.700542 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.699171    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5dzh\" (UniqueName: \"kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-kube-api-access-h5dzh\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.700542 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.699216    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-token\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.700542 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.699237    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8cf4110-29ff-414d-87af-dfa343a91bc7-istiod-ca-cert\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.700542 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.699300    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-credential-socket\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.700542 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.699399    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-certs\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.710506 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.710417    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.713212 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.712665    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.713212 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.712962    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.766626 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.766500    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:17.766626 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.766557    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.790399 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.777544    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e74c27-openshift-default-dockercfg-x86sb\""
Oct 30 13:00:17.819544 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.802641    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5dzh\" (UniqueName: \"kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-kube-api-access-h5dzh\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.819544 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.802662    4522 patch_prober.go:29] interesting pod/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:17.819544 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.802716    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-token\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.819544 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.802745    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8cf4110-29ff-414d-87af-dfa343a91bc7-istiod-ca-cert\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.819544 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.802824    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-credential-socket\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.819544 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.802918    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-certs\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.819544 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.802978    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-podinfo\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.819544 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.803007    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-data\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.819544 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.803059    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-socket\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.819544 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.803113    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-envoy\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.829898 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.829162    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8cf4110-29ff-414d-87af-dfa343a91bc7-istiod-ca-cert\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.829898 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.802736    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:17.843388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.830369    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-credential-socket\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.843388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.830646    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-certs\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.843388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.830909    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-data\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.843388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.831170    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-socket\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.865038 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.864872    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-envoy\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.900505 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.900422    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-podinfo\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.939445 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.901308    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5dzh\" (UniqueName: \"kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-kube-api-access-h5dzh\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:17.941316 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:17.941250    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-token\") pod \"e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") " pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:18.091539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.091489    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:18.091846 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.091815    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:18.165907 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.165872    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:00:18.166424 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.166313    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:00:18.167210 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.167141    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:18.235372 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.232469    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:18.235372 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.232849    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:00:18.235372 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.233110    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:00:18.235372 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.233373    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:00:18.239374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.236759    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:18.239374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.237124    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:18.336526 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.336477    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:00:18.719026 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.718978    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:18.719459 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.719418    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:18.872780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.864048    4522 generic.go:356] "Generic (PLEG): container finished" podID="5108bd61-b167-45ee-88f3-361559eab957" containerID="d5180d2ef1db26f65c6eba1424d1c6590304fc0dd2bb658a9f69c47bfa31a5bf" exitCode=60
Oct 30 13:00:18.872780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.864231    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd" event={"ID":"5108bd61-b167-45ee-88f3-361559eab957","Type":"ContainerDied","Data":"d5180d2ef1db26f65c6eba1424d1c6590304fc0dd2bb658a9f69c47bfa31a5bf"}
Oct 30 13:00:18.872780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.864295    4522 scope.go:122] "RemoveContainer" containerID="4511bf3b4c2a1a8295f1752420449b0b3d05fa89758a95294d2c79e6bcb74496"
Oct 30 13:00:18.872780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:18.865043    4522 scope.go:122] "RemoveContainer" containerID="d5180d2ef1db26f65c6eba1424d1c6590304fc0dd2bb658a9f69c47bfa31a5bf"
Oct 30 13:00:18.872780 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:18.865524    4522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd_odh-ai-gateway-infra(5108bd61-b167-45ee-88f3-361559eab957)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd" podUID="5108bd61-b167-45ee-88f3-361559eab957"
Oct 30 13:00:19.154492 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.123677    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th" event={"ID":"87ca2fcf-24d8-409c-b345-c20dc74cbb34","Type":"ContainerStarted","Data":"ad24476442731d53a81586404d9315195bae97e0a2f64b3742cef7f07c863668"}
Oct 30 13:00:19.154492 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.133711    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7" event={"ID":"645c0ee8-5706-4071-a097-22f266a84e55","Type":"ContainerStarted","Data":"779109fe6cf5dd6b20b6fb99a81b2272bd631740303c3f115c38eda4084cf309"}
Oct 30 13:00:19.179294 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.167815    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:19.179294 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.169140    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:19.237393 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.237169    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th" podStartSLOduration=19.237147047 podStartE2EDuration="19.237147047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:19.20747938 +0000 UTC m=+1463.022805895" watchObservedRunningTime="2026-10-30 13:00:19.237147047 +0000 UTC m=+1463.052473537"
Oct 30 13:00:19.293572 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.261472    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:00:19.293572 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.264641    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:00:19.293572 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.264866    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:00:19.293572 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.265037    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:00:19.293572 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.265183    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:00:19.346477 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.299674    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:00:19.393423 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.368486    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:00:19.616929 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.552179    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:19.779379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.752543    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:00:19.802408 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:19.782885    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:20.037699 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.037642    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:20.038297 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.038266    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:20.038539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.038513    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:20.192569 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.192460    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:20.253858 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.196504    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:00:20.270813 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.262171    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:20.272809 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.272759    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:20.290468 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.273212    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:20.291256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.291222    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:20.291860 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.291822    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:20.298657 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.298576    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7" podStartSLOduration=20.298557941 podStartE2EDuration="20.298557941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:19.714497324 +0000 UTC m=+1463.529823815" watchObservedRunningTime="2026-10-30 13:00:20.298557941 +0000 UTC m=+1464.113884442"
Oct 30 13:00:20.392741 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.392691    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:20.393943 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.393904    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:20.418416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.418375    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:20.418755 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.418723    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:20.799261 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.772107    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded"
Oct 30 13:00:21.421555 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:20.956013    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:00:21.443373 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.443317    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:00:21.452854 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.036623    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:21.452854 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.115883    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:00:21.456481 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.198326    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 30 13:00:21.502632 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.035524    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:21.542905 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.542862    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:21.669436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.275892    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:21.669436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.275978    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:21.669436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.304433    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:21.669436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.304475    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:00:21.669436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.304512    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:00:21.695786 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.669963    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:21.695786 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.275554    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:00:21.695786 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.275594    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:00:21.695786 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.502844    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:21.695786 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:21.588151    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:00:21.695786 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.248034    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:00:21.695786 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.692928    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:00:21.810821 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.810784    4522 generic.go:356] "Generic (PLEG): container finished" podID="87ca2fcf-24d8-409c-b345-c20dc74cbb34" containerID="ad24476442731d53a81586404d9315195bae97e0a2f64b3742cef7f07c863668" exitCode=60
Oct 30 13:00:21.915755 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:21.915699    4522 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.558s"
Oct 30 13:00:21.929393 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.929298    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"]
Oct 30 13:00:21.931854 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.931837    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:00:21.932028 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.932009    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" event={"ID":"a8cf4110-29ff-414d-87af-dfa343a91bc7","Type":"ContainerStarted","Data":"787db61112401b81ff9f7dfc40a08cc4faa97b1b7c55218947945437024ad337"}
Oct 30 13:00:21.932149 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.932136    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"]
Oct 30 13:00:21.932264 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.932252    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th" event={"ID":"87ca2fcf-24d8-409c-b345-c20dc74cbb34","Type":"ContainerDied","Data":"ad24476442731d53a81586404d9315195bae97e0a2f64b3742cef7f07c863668"}
Oct 30 13:00:21.932419 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.932407    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:00:21.932536 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.932526    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"]
Oct 30 13:00:21.956241 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:21.956219    4522 scope.go:122] "RemoveContainer" containerID="84647a694baa373d405cf934a4a1f0ff0158a35fe617d1850d2c9a0f242f4408"
Oct 30 13:00:22.052384 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.048511    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"]
Oct 30 13:00:22.052384 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.048616    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:00:22.052384 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.048653    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:00:22.052384 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.048721    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:00:22.052384 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.049527    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7" podUID="645c0ee8-5706-4071-a097-22f266a84e55" containerName="cleanup" containerID="cri-o://779109fe6cf5dd6b20b6fb99a81b2272bd631740303c3f115c38eda4084cf309" gracePeriod=30
Oct 30 13:00:22.056045 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.055841    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:00:22.063192 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.062501    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:00:22.067372 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.067308    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:00:22.165196 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.165169    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:00:22.203431 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.177773    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 13:00:22.227373 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.224414    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:00:22.811937 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.794569    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"
Oct 30 13:00:22.811937 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.808460    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5108bd61-b167-45ee-88f3-361559eab957-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5108bd61-b167-45ee-88f3-361559eab957-maas-api-service-ca\") pod \"5108bd61-b167-45ee-88f3-361559eab957\" (UID: \"5108bd61-b167-45ee-88f3-361559eab957\") "
Oct 30 13:00:22.811937 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.808556    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5108bd61-b167-45ee-88f3-361559eab957-kube-api-access-7jjbg\" (UniqueName: \"kubernetes.io/projected/5108bd61-b167-45ee-88f3-361559eab957-kube-api-access-7jjbg\") pod \"5108bd61-b167-45ee-88f3-361559eab957\" (UID: \"5108bd61-b167-45ee-88f3-361559eab957\") "
Oct 30 13:00:22.878420 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.820500    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5108bd61-b167-45ee-88f3-361559eab957-kube-api-access-7jjbg" pod "5108bd61-b167-45ee-88f3-361559eab957" (UID: "5108bd61-b167-45ee-88f3-361559eab957"). InnerVolumeSpecName "kube-api-access-7jjbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:22.878420 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.821049    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5108bd61-b167-45ee-88f3-361559eab957-maas-api-service-ca" pod "5108bd61-b167-45ee-88f3-361559eab957" (UID: "5108bd61-b167-45ee-88f3-361559eab957"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:22.878420 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.871584    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"
Oct 30 13:00:22.914450 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.914396    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5108bd61-b167-45ee-88f3-361559eab957-maas-api-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.914685 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.914663    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jjbg\" (UniqueName: \"kubernetes.io/projected/5108bd61-b167-45ee-88f3-361559eab957-kube-api-access-7jjbg\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.988582 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.988551    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th" event={"ID":"87ca2fcf-24d8-409c-b345-c20dc74cbb34","Type":"ContainerDied","Data":"e12d06bc938a1ae791188c6ad85f07cf0312027fb8f2aa2f2500fefdc079d4b8"}
Oct 30 13:00:22.988787 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.988772    4522 scope.go:122] "RemoveContainer" containerID="ad24476442731d53a81586404d9315195bae97e0a2f64b3742cef7f07c863668"
Oct 30 13:00:22.993174 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.993147    4522 generic.go:356] "Generic (PLEG): container finished" podID="645c0ee8-5706-4071-a097-22f266a84e55" containerID="779109fe6cf5dd6b20b6fb99a81b2272bd631740303c3f115c38eda4084cf309" exitCode=60
Oct 30 13:00:22.993576 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.993551    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7" event={"ID":"645c0ee8-5706-4071-a097-22f266a84e55","Type":"ContainerDied","Data":"779109fe6cf5dd6b20b6fb99a81b2272bd631740303c3f115c38eda4084cf309"}
Oct 30 13:00:22.995511 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.995488    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd" event={"ID":"5108bd61-b167-45ee-88f3-361559eab957","Type":"ContainerDied","Data":"0bb616b24e1c66d28e434b9d494c0108374c7ff39d53959b36aa93f253494242"}
Oct 30 13:00:22.996048 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:22.996030    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"
Oct 30 13:00:23.005766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.005738    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" event={"ID":"a8cf4110-29ff-414d-87af-dfa343a91bc7","Type":"ContainerStarted","Data":"5aa13ef721e419998aaf634c1f9571846d15e9d37bbd5fdb8598c877335f43ed"}
Oct 30 13:00:23.020403 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.019375    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ca2fcf-24d8-409c-b345-c20dc74cbb34-kube-api-access-4xrlm\" (UniqueName: \"kubernetes.io/projected/87ca2fcf-24d8-409c-b345-c20dc74cbb34-kube-api-access-4xrlm\") pod \"87ca2fcf-24d8-409c-b345-c20dc74cbb34\" (UID: \"87ca2fcf-24d8-409c-b345-c20dc74cbb34\") "
Oct 30 13:00:23.020403 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.019444    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ca2fcf-24d8-409c-b345-c20dc74cbb34-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ca2fcf-24d8-409c-b345-c20dc74cbb34-maas-api-service-ca\") pod \"87ca2fcf-24d8-409c-b345-c20dc74cbb34\" (UID: \"87ca2fcf-24d8-409c-b345-c20dc74cbb34\") "
Oct 30 13:00:23.023277 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.023202    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ca2fcf-24d8-409c-b345-c20dc74cbb34-maas-api-service-ca" pod "87ca2fcf-24d8-409c-b345-c20dc74cbb34" (UID: "87ca2fcf-24d8-409c-b345-c20dc74cbb34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:23.024316 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.024171    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ca2fcf-24d8-409c-b345-c20dc74cbb34-kube-api-access-4xrlm" pod "87ca2fcf-24d8-409c-b345-c20dc74cbb34" (UID: "87ca2fcf-24d8-409c-b345-c20dc74cbb34"). InnerVolumeSpecName "kube-api-access-4xrlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:23.042626 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.042598    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"
Oct 30 13:00:23.042874 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.042852    4522 scope.go:122] "RemoveContainer" containerID="0f8569816af6789dae6b8809a0eb3327df5f36bf24c332b92013d73df22f4369"
Oct 30 13:00:23.060508 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.060463    4522 scope.go:122] "RemoveContainer" containerID="d5180d2ef1db26f65c6eba1424d1c6590304fc0dd2bb658a9f69c47bfa31a5bf"
Oct 30 13:00:23.126516 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.123083    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xrlm\" (UniqueName: \"kubernetes.io/projected/87ca2fcf-24d8-409c-b345-c20dc74cbb34-kube-api-access-4xrlm\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.126516 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.123121    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ca2fcf-24d8-409c-b345-c20dc74cbb34-maas-api-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.190686 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.169517    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"]
Oct 30 13:00:23.190686 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.169583    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:23.190686 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.180639    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:23.211592 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.210479    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c81f42-29889420-mzxfd"]
Oct 30 13:00:23.222462 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.222386    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" podStartSLOduration=8.222365186 podStartE2EDuration="8.222365186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:23.208388405 +0000 UTC m=+1467.023714895" watchObservedRunningTime="2026-10-30 13:00:23.222365186 +0000 UTC m=+1467.037691673"
Oct 30 13:00:23.227385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.225183    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645c0ee8-5706-4071-a097-22f266a84e55-kube-api-access-6cnrb\" (UniqueName: \"kubernetes.io/projected/645c0ee8-5706-4071-a097-22f266a84e55-kube-api-access-6cnrb\") pod \"645c0ee8-5706-4071-a097-22f266a84e55\" (UID: \"645c0ee8-5706-4071-a097-22f266a84e55\") "
Oct 30 13:00:23.227385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.225394    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645c0ee8-5706-4071-a097-22f266a84e55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645c0ee8-5706-4071-a097-22f266a84e55-maas-api-service-ca\") pod \"645c0ee8-5706-4071-a097-22f266a84e55\" (UID: \"645c0ee8-5706-4071-a097-22f266a84e55\") "
Oct 30 13:00:23.231386 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.230640    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:23.231386 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.231249    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645c0ee8-5706-4071-a097-22f266a84e55-maas-api-service-ca" pod "645c0ee8-5706-4071-a097-22f266a84e55" (UID: "645c0ee8-5706-4071-a097-22f266a84e55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:23.307527 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.306571    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645c0ee8-5706-4071-a097-22f266a84e55-kube-api-access-6cnrb" pod "645c0ee8-5706-4071-a097-22f266a84e55" (UID: "645c0ee8-5706-4071-a097-22f266a84e55"). InnerVolumeSpecName "kube-api-access-6cnrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:23.328002 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.327919    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cnrb\" (UniqueName: \"kubernetes.io/projected/645c0ee8-5706-4071-a097-22f266a84e55-kube-api-access-6cnrb\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.328002 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.327953    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645c0ee8-5706-4071-a097-22f266a84e55-maas-api-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.352248 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.352210    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:23.415171 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.415137    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 13:00:23.429991 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.429932    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0519800-4332-4a1c-94a8-205367079050-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0519800-4332-4a1c-94a8-205367079050-istio-podinfo\") pod \"b0519800-4332-4a1c-94a8-205367079050\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") "
Oct 30 13:00:23.429991 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.429975    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-socket\") pod \"b0519800-4332-4a1c-94a8-205367079050\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") "
Oct 30 13:00:23.430173 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.430000    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0519800-4332-4a1c-94a8-205367079050-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0519800-4332-4a1c-94a8-205367079050-istiod-ca-cert\") pod \"b0519800-4332-4a1c-94a8-205367079050\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") "
Oct 30 13:00:23.430173 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.430100    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-certs\") pod \"b0519800-4332-4a1c-94a8-205367079050\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") "
Oct 30 13:00:23.430348 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.430188    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-credential-socket\") pod \"b0519800-4332-4a1c-94a8-205367079050\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") "
Oct 30 13:00:23.430348 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.430318    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-envoy\") pod \"b0519800-4332-4a1c-94a8-205367079050\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") "
Oct 30 13:00:23.430539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.430403    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-istio-token\" (UniqueName: \"kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-istio-token\") pod \"b0519800-4332-4a1c-94a8-205367079050\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") "
Oct 30 13:00:23.430539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.430431    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-kube-api-access-tv7v5\" (UniqueName: \"kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-kube-api-access-tv7v5\") pod \"b0519800-4332-4a1c-94a8-205367079050\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") "
Oct 30 13:00:23.430694 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.430563    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-data\") pod \"b0519800-4332-4a1c-94a8-205367079050\" (UID: \"b0519800-4332-4a1c-94a8-205367079050\") "
Oct 30 13:00:23.445572 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.435715    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-data" pod "b0519800-4332-4a1c-94a8-205367079050" (UID: "b0519800-4332-4a1c-94a8-205367079050"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:23.445572 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.436078    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-socket" pod "b0519800-4332-4a1c-94a8-205367079050" (UID: "b0519800-4332-4a1c-94a8-205367079050"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:23.445572 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.440541    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0519800-4332-4a1c-94a8-205367079050-istiod-ca-cert" pod "b0519800-4332-4a1c-94a8-205367079050" (UID: "b0519800-4332-4a1c-94a8-205367079050"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:23.498471 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.498426    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-credential-socket" pod "b0519800-4332-4a1c-94a8-205367079050" (UID: "b0519800-4332-4a1c-94a8-205367079050"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:23.498915 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.498886    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-certs" pod "b0519800-4332-4a1c-94a8-205367079050" (UID: "b0519800-4332-4a1c-94a8-205367079050"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:23.575833 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.536754    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:23.575833 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.538801    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.575833 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.538829    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.575833 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.538844    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0519800-4332-4a1c-94a8-205367079050-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.575833 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.538861    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.575833 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.538874    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.615855 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.615692    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-istio-token" pod "b0519800-4332-4a1c-94a8-205367079050" (UID: "b0519800-4332-4a1c-94a8-205367079050"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:23.615855 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.615814    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0519800-4332-4a1c-94a8-205367079050-istio-podinfo" pod "b0519800-4332-4a1c-94a8-205367079050" (UID: "b0519800-4332-4a1c-94a8-205367079050"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:00:23.649376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.637898    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-kube-api-access-tv7v5" pod "b0519800-4332-4a1c-94a8-205367079050" (UID: "b0519800-4332-4a1c-94a8-205367079050"). InnerVolumeSpecName "kube-api-access-tv7v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:23.649376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.649130    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.649376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.649163    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv7v5\" (UniqueName: \"kubernetes.io/projected/b0519800-4332-4a1c-94a8-205367079050-kube-api-access-tv7v5\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.649376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.649182    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0519800-4332-4a1c-94a8-205367079050-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.676215 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.657956    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-envoy" pod "b0519800-4332-4a1c-94a8-205367079050" (UID: "b0519800-4332-4a1c-94a8-205367079050"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:23.712376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.702693    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:23.807377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:23.795710    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0519800-4332-4a1c-94a8-205367079050-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:24.015662 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.015558    4522 generic.go:356] "Generic (PLEG): container finished" podID="b0519800-4332-4a1c-94a8-205367079050" containerID="2b2a584aa0dba1b26666e4275189b70c39f088c4d05c2ac8e5e3e130ca4f915a" exitCode=0
Oct 30 13:00:24.015662 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.015612    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"
Oct 30 13:00:24.015662 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.015640    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6" event={"ID":"b0519800-4332-4a1c-94a8-205367079050","Type":"ContainerDied","Data":"2b2a584aa0dba1b26666e4275189b70c39f088c4d05c2ac8e5e3e130ca4f915a"}
Oct 30 13:00:24.016255 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.015703    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6" event={"ID":"b0519800-4332-4a1c-94a8-205367079050","Type":"ContainerDied","Data":"cd6c26d8e039b9c169ecc5aa8ee53faed42ff9ca9828e4c4da93034ad0d03d50"}
Oct 30 13:00:24.016255 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.015735    4522 scope.go:122] "RemoveContainer" containerID="2b2a584aa0dba1b26666e4275189b70c39f088c4d05c2ac8e5e3e130ca4f915a"
Oct 30 13:00:24.016686 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.016665    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"
Oct 30 13:00:24.019275 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.019244    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"
Oct 30 13:00:24.019403 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.019275    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7" event={"ID":"645c0ee8-5706-4071-a097-22f266a84e55","Type":"ContainerDied","Data":"7aa803586ea070df4e4012ef907d19c0215efa970bb8ea5f72812184c8d8e35f"}
Oct 30 13:00:24.020088 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.020067    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:24.025299 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.025274    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:00:24.027789 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.027477    4522 scope.go:122] "RemoveContainer" containerID="2b2a584aa0dba1b26666e4275189b70c39f088c4d05c2ac8e5e3e130ca4f915a"
Oct 30 13:00:24.028037 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:24.027983    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2a584aa0dba1b26666e4275189b70c39f088c4d05c2ac8e5e3e130ca4f915a\": container with ID starting with 2b2a584aa0dba1b26666e4275189b70c39f088c4d05c2ac8e5e3e130ca4f915a not found: ID does not exist" containerID="2b2a584aa0dba1b26666e4275189b70c39f088c4d05c2ac8e5e3e130ca4f915a"
Oct 30 13:00:24.028037 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.028020    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2a584aa0dba1b26666e4275189b70c39f088c4d05c2ac8e5e3e130ca4f915a"} err="failed to get container status \"2b2a584aa0dba1b26666e4275189b70c39f088c4d05c2ac8e5e3e130ca4f915a\": rpc error: code = NotFound desc = could not find container \"2b2a584aa0dba1b26666e4275189b70c39f088c4d05c2ac8e5e3e130ca4f915a\": container with ID starting with 2b2a584aa0dba1b26666e4275189b70c39f088c4d05c2ac8e5e3e130ca4f915a not found: ID does not exist"
Oct 30 13:00:24.028157 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.028046    4522 scope.go:122] "RemoveContainer" containerID="779109fe6cf5dd6b20b6fb99a81b2272bd631740303c3f115c38eda4084cf309"
Oct 30 13:00:24.263065 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.262040    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"]
Oct 30 13:00:24.306124 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.306093    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-vw4th"]
Oct 30 13:00:24.310317 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.310296    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"]
Oct 30 13:00:24.315443 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.315420    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7a04e002-openshift-default-6669855c89-9khb6"]
Oct 30 13:00:24.325242 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.325218    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"]
Oct 30 13:00:24.334889 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.334865    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5108bd61-b167-45ee-88f3-361559eab957" path="/var/lib/kubelet/pods/5108bd61-b167-45ee-88f3-361559eab957/volumes"
Oct 30 13:00:24.335339 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.335325    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ca2fcf-24d8-409c-b345-c20dc74cbb34" path="/var/lib/kubelet/pods/87ca2fcf-24d8-409c-b345-c20dc74cbb34/volumes"
Oct 30 13:00:24.335841 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.335827    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0519800-4332-4a1c-94a8-205367079050" path="/var/lib/kubelet/pods/b0519800-4332-4a1c-94a8-205367079050/volumes"
Oct 30 13:00:24.336452 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:24.336437    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2bfb1b-29889420-6cvs7"]
Oct 30 13:00:25.348881 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:25.348838    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:25.826997 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:25.826955    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:25.929941 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:25.929900    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:26.204057 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:26.203970    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:26.337299 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:26.337265    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645c0ee8-5706-4071-a097-22f266a84e55" path="/var/lib/kubelet/pods/645c0ee8-5706-4071-a097-22f266a84e55/volumes"
Oct 30 13:00:27.533229 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.533177    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:27.630304 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.630263    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:27.709142 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.709109    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"]
Oct 30 13:00:27.710095 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710073    4522 cpu_manager.go:414] "containerMap: removing container" podUID="5108bd61-b167-45ee-88f3-361559eab957" containerName="cleanup"
Oct 30 13:00:27.710153 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710101    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5108bd61-b167-45ee-88f3-361559eab957" containerName="cleanup"
Oct 30 13:00:27.710153 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710129    4522 cpu_manager.go:414] "containerMap: removing container" podUID="645c0ee8-5706-4071-a097-22f266a84e55" containerName="cleanup"
Oct 30 13:00:27.710153 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710136    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645c0ee8-5706-4071-a097-22f266a84e55" containerName="cleanup"
Oct 30 13:00:27.710153 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710151    4522 cpu_manager.go:414] "containerMap: removing container" podUID="645c0ee8-5706-4071-a097-22f266a84e55" containerName="cleanup"
Oct 30 13:00:27.710281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710158    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645c0ee8-5706-4071-a097-22f266a84e55" containerName="cleanup"
Oct 30 13:00:27.710281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710169    4522 cpu_manager.go:414] "containerMap: removing container" podUID="645c0ee8-5706-4071-a097-22f266a84e55" containerName="cleanup"
Oct 30 13:00:27.710281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710177    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645c0ee8-5706-4071-a097-22f266a84e55" containerName="cleanup"
Oct 30 13:00:27.710281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710190    4522 cpu_manager.go:414] "containerMap: removing container" podUID="b0519800-4332-4a1c-94a8-205367079050" containerName="istio-proxy"
Oct 30 13:00:27.710281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710196    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0519800-4332-4a1c-94a8-205367079050" containerName="istio-proxy"
Oct 30 13:00:27.710281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710212    4522 cpu_manager.go:414] "containerMap: removing container" podUID="87ca2fcf-24d8-409c-b345-c20dc74cbb34" containerName="cleanup"
Oct 30 13:00:27.710281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710219    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ca2fcf-24d8-409c-b345-c20dc74cbb34" containerName="cleanup"
Oct 30 13:00:27.710281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710238    4522 cpu_manager.go:414] "containerMap: removing container" podUID="5108bd61-b167-45ee-88f3-361559eab957" containerName="cleanup"
Oct 30 13:00:27.710281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710245    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5108bd61-b167-45ee-88f3-361559eab957" containerName="cleanup"
Oct 30 13:00:27.710281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710270    4522 cpu_manager.go:414] "containerMap: removing container" podUID="87ca2fcf-24d8-409c-b345-c20dc74cbb34" containerName="cleanup"
Oct 30 13:00:27.710281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710277    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ca2fcf-24d8-409c-b345-c20dc74cbb34" containerName="cleanup"
Oct 30 13:00:27.710651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710289    4522 cpu_manager.go:414] "containerMap: removing container" podUID="5108bd61-b167-45ee-88f3-361559eab957" containerName="cleanup"
Oct 30 13:00:27.710651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710296    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5108bd61-b167-45ee-88f3-361559eab957" containerName="cleanup"
Oct 30 13:00:27.710651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710321    4522 cpu_manager.go:414] "containerMap: removing container" podUID="87ca2fcf-24d8-409c-b345-c20dc74cbb34" containerName="cleanup"
Oct 30 13:00:27.710651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710328    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ca2fcf-24d8-409c-b345-c20dc74cbb34" containerName="cleanup"
Oct 30 13:00:27.710651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710504    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5108bd61-b167-45ee-88f3-361559eab957" containerName="cleanup"
Oct 30 13:00:27.710651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710523    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5108bd61-b167-45ee-88f3-361559eab957" containerName="cleanup"
Oct 30 13:00:27.710651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710532    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ca2fcf-24d8-409c-b345-c20dc74cbb34" containerName="cleanup"
Oct 30 13:00:27.710651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710543    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ca2fcf-24d8-409c-b345-c20dc74cbb34" containerName="cleanup"
Oct 30 13:00:27.710651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710558    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="645c0ee8-5706-4071-a097-22f266a84e55" containerName="cleanup"
Oct 30 13:00:27.710651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710569    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="645c0ee8-5706-4071-a097-22f266a84e55" containerName="cleanup"
Oct 30 13:00:27.710651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710578    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="645c0ee8-5706-4071-a097-22f266a84e55" containerName="cleanup"
Oct 30 13:00:27.710651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.710588    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0519800-4332-4a1c-94a8-205367079050" containerName="istio-proxy"
Oct 30 13:00:27.714788 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.714765    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"
Oct 30 13:00:27.729726 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.729698    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"]
Oct 30 13:00:27.734316 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.734290    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e74c27\""
Oct 30 13:00:27.743529 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.739979    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e74c27-dockercfg-bdvx5\""
Oct 30 13:00:27.910428 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.831465    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfll4\" (UniqueName: \"kubernetes.io/projected/796b8883-18c7-4535-bc2a-eb8b09f78e9e-kube-api-access-rfll4\") pod \"payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv\" (UID: \"796b8883-18c7-4535-bc2a-eb8b09f78e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"
Oct 30 13:00:27.910428 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.831817    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796b8883-18c7-4535-bc2a-eb8b09f78e9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv\" (UID: \"796b8883-18c7-4535-bc2a-eb8b09f78e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"
Oct 30 13:00:27.943424 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.941327    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfll4\" (UniqueName: \"kubernetes.io/projected/796b8883-18c7-4535-bc2a-eb8b09f78e9e-kube-api-access-rfll4\") pod \"payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv\" (UID: \"796b8883-18c7-4535-bc2a-eb8b09f78e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"
Oct 30 13:00:27.943424 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.941761    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796b8883-18c7-4535-bc2a-eb8b09f78e9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv\" (UID: \"796b8883-18c7-4535-bc2a-eb8b09f78e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"
Oct 30 13:00:27.943424 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:27.942717    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796b8883-18c7-4535-bc2a-eb8b09f78e9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv\" (UID: \"796b8883-18c7-4535-bc2a-eb8b09f78e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"
Oct 30 13:00:28.052778 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.048181    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfll4\" (UniqueName: \"kubernetes.io/projected/796b8883-18c7-4535-bc2a-eb8b09f78e9e-kube-api-access-rfll4\") pod \"payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv\" (UID: \"796b8883-18c7-4535-bc2a-eb8b09f78e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"
Oct 30 13:00:28.090515 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.090470    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"]
Oct 30 13:00:28.109470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.109422    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"
Oct 30 13:00:28.111347 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.111317    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="5108bd61-b167-45ee-88f3-361559eab957" containerName="cleanup"
Oct 30 13:00:28.111636 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.111620    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ca2fcf-24d8-409c-b345-c20dc74cbb34" containerName="cleanup"
Oct 30 13:00:28.151707 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.151655    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"]
Oct 30 13:00:28.152125 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.152110    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"
Oct 30 13:00:28.261007 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.260971    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c14230-554c-403a-b08b-88db63161d71-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn\" (UID: \"19c14230-554c-403a-b08b-88db63161d71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"
Oct 30 13:00:28.261189 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.261173    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr5vp\" (UniqueName: \"kubernetes.io/projected/19c14230-554c-403a-b08b-88db63161d71-kube-api-access-lr5vp\") pod \"payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn\" (UID: \"19c14230-554c-403a-b08b-88db63161d71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"
Oct 30 13:00:28.362902 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.362860    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr5vp\" (UniqueName: \"kubernetes.io/projected/19c14230-554c-403a-b08b-88db63161d71-kube-api-access-lr5vp\") pod \"payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn\" (UID: \"19c14230-554c-403a-b08b-88db63161d71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"
Oct 30 13:00:28.363113 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.363059    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c14230-554c-403a-b08b-88db63161d71-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn\" (UID: \"19c14230-554c-403a-b08b-88db63161d71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"
Oct 30 13:00:28.363997 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.363967    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c14230-554c-403a-b08b-88db63161d71-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn\" (UID: \"19c14230-554c-403a-b08b-88db63161d71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"
Oct 30 13:00:28.381549 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.381520    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr5vp\" (UniqueName: \"kubernetes.io/projected/19c14230-554c-403a-b08b-88db63161d71-kube-api-access-lr5vp\") pod \"payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn\" (UID: \"19c14230-554c-403a-b08b-88db63161d71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"
Oct 30 13:00:28.470833 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:00:28.470805    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796b8883_18c7_4535_bc2a_eb8b09f78e9e.slice/crio-678f4397e3f265a48a3e08798f6da6d1ae73e9be9fc9d1e04463cd72e3285cf2 WatchSource:0}: Error finding container 678f4397e3f265a48a3e08798f6da6d1ae73e9be9fc9d1e04463cd72e3285cf2: Status 404 returned error can't find the container with id 678f4397e3f265a48a3e08798f6da6d1ae73e9be9fc9d1e04463cd72e3285cf2
Oct 30 13:00:28.471818 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.471790    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"]
Oct 30 13:00:28.473876 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.473828    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"
Oct 30 13:00:28.787039 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:00:28.787009    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c14230_554c_403a_b08b_88db63161d71.slice/crio-ac4a2ff95de3e1f2f55b92edaf25950dff4a5c22d12fed53642ee0563dcab247 WatchSource:0}: Error finding container ac4a2ff95de3e1f2f55b92edaf25950dff4a5c22d12fed53642ee0563dcab247: Status 404 returned error can't find the container with id ac4a2ff95de3e1f2f55b92edaf25950dff4a5c22d12fed53642ee0563dcab247
Oct 30 13:00:28.787696 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:28.787627    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"]
Oct 30 13:00:29.156422 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:29.156299    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn" event={"ID":"19c14230-554c-403a-b08b-88db63161d71","Type":"ContainerStarted","Data":"a74425809d967a093e3243170d51be940f43e0c2a2ff2f81d1ae835401196422"}
Oct 30 13:00:29.156422 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:29.156367    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn" event={"ID":"19c14230-554c-403a-b08b-88db63161d71","Type":"ContainerStarted","Data":"ac4a2ff95de3e1f2f55b92edaf25950dff4a5c22d12fed53642ee0563dcab247"}
Oct 30 13:00:29.156422 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:29.156397    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"
Oct 30 13:00:29.158107 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:29.158083    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv" event={"ID":"796b8883-18c7-4535-bc2a-eb8b09f78e9e","Type":"ContainerStarted","Data":"ae7032ae1db4ce5d0d19c2d615693c91ad39fa7286823c8f2aeb89553076ad69"}
Oct 30 13:00:29.158260 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:29.158118    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv" event={"ID":"796b8883-18c7-4535-bc2a-eb8b09f78e9e","Type":"ContainerStarted","Data":"678f4397e3f265a48a3e08798f6da6d1ae73e9be9fc9d1e04463cd72e3285cf2"}
Oct 30 13:00:29.158260 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:29.158138    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"
Oct 30 13:00:29.197549 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:29.197489    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn" podStartSLOduration=2.197474537 podStartE2EDuration="2.197474537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:29.194546659 +0000 UTC m=+1473.009873144" watchObservedRunningTime="2026-10-30 13:00:29.197474537 +0000 UTC m=+1473.012801058"
Oct 30 13:00:29.234718 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:29.234652    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv" podStartSLOduration=2.234637321 podStartE2EDuration="2.234637321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:29.231686271 +0000 UTC m=+1473.047012760" watchObservedRunningTime="2026-10-30 13:00:29.234637321 +0000 UTC m=+1473.049963809"
Oct 30 13:00:29.597713 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:29.597673    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:30.166089 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:30.165999    4522 generic.go:356] "Generic (PLEG): container finished" podID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerID="fc9886805ac5845ffdd9e65fc024783e920d62af173d339af8e96bb9a8035e26" exitCode=0
Oct 30 13:00:30.166606 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:30.166077    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" event={"ID":"54c66945-0e5f-44d3-b5ec-228ddc468ffb","Type":"ContainerDied","Data":"fc9886805ac5845ffdd9e65fc024783e920d62af173d339af8e96bb9a8035e26"}
Oct 30 13:00:30.166773 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:30.166756    4522 scope.go:122] "RemoveContainer" containerID="fc9886805ac5845ffdd9e65fc024783e920d62af173d339af8e96bb9a8035e26"
Oct 30 13:00:30.168219 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:30.168191    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:30.168288 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:30.168260    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:30.168328 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:30.168292    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:31.236390 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:31.232039    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" event={"ID":"54c66945-0e5f-44d3-b5ec-228ddc468ffb","Type":"ContainerStarted","Data":"eeac5e62da8ca49b63d8cacc6757c86daf70d6e6168216ea8b84f2e041afd37a"}
Oct 30 13:00:31.376377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:31.370557    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 13:00:31.376377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:31.372925    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 13:00:31.376377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:31.372985    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 13:00:31.901193 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:31.901160    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:32.350851 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:32.350821    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 13:00:32.351266 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:32.350870    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 13:00:33.351387 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:33.351326    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 13:00:33.351807 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:33.351416    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 13:00:33.393884 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:33.393844    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:33.594871 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:33.594818    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:33.692842 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:33.680828    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:33.959432 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:33.952828    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:34.074294 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:34.074241    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:34.202266 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:34.198577    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:34.353203 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:34.353165    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 13:00:34.354031 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:34.353995    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 13:00:34.603256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:34.603164    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:35.350914 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:35.350875    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 13:00:35.351100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:35.350946    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 13:00:35.489909 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:35.489873    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:36.253808 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:36.251684    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:36.408372 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:36.400000    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 13:00:36.408372 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:36.400097    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 13:00:37.355847 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:37.355817    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 13:00:37.356313 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:37.356296    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 13:00:37.356922 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:37.356903    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 13:00:37.933693 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:37.933656    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"]
Oct 30 13:00:37.970482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:37.970444    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"]
Oct 30 13:00:38.041587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:38.041546    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:38.126111 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:38.126067    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:38.242134 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:38.242029    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:40.168740 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:40.168705    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"
Oct 30 13:00:40.169227 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:40.168784    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"
Oct 30 13:00:40.419980 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:40.419905    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:40.627261 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:40.627228    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:40.770929 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:40.770888    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"]
Oct 30 13:00:40.771149 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:40.771125    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc" podUID="d8424038-b3b1-4839-9bc2-50ee8ef95419" containerName="istio-proxy" containerID="cri-o://9557a40c3d0ff700121421fef27f3027a8c7fb55149cd3c287a28a457b9bfff0" gracePeriod=30
Oct 30 13:00:40.799811 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:40.799783    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:41.902288 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:41.902252    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:42.226905 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:42.226807    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:42.934069 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:42.934027    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn" podUID="b4a81a9a-e4ef-453a-8de4-96b9f403b134" containerName="payload-processing" containerID="cri-o://0cb9cab5d374374ae8a39122926680ed8441d7219d8970713c3326f92a13252c" gracePeriod=25
Oct 30 13:00:42.971002 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:42.970961    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7" podUID="6936032a-d78e-4d02-bb45-358368095ecf" containerName="payload-pre-processing" containerID="cri-o://8cfc99443dc24e6ba6cebe35e523de064eea80aac582b870f1ee9573db09cdff" gracePeriod=25
Oct 30 13:00:43.281188 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.281155    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"
Oct 30 13:00:43.320014 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.319979    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"
Oct 30 13:00:43.370152 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.370120    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6936032a-d78e-4d02-bb45-358368095ecf-kube-api-access-ltf45\" (UniqueName: \"kubernetes.io/projected/6936032a-d78e-4d02-bb45-358368095ecf-kube-api-access-ltf45\") pod \"6936032a-d78e-4d02-bb45-358368095ecf\" (UID: \"6936032a-d78e-4d02-bb45-358368095ecf\") "
Oct 30 13:00:43.370392 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.370250    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a81a9a-e4ef-453a-8de4-96b9f403b134-kube-api-access-b4jnf\" (UniqueName: \"kubernetes.io/projected/b4a81a9a-e4ef-453a-8de4-96b9f403b134-kube-api-access-b4jnf\") pod \"b4a81a9a-e4ef-453a-8de4-96b9f403b134\" (UID: \"b4a81a9a-e4ef-453a-8de4-96b9f403b134\") "
Oct 30 13:00:43.370392 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.370285    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6936032a-d78e-4d02-bb45-358368095ecf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6936032a-d78e-4d02-bb45-358368095ecf-plugins-config-volume\") pod \"6936032a-d78e-4d02-bb45-358368095ecf\" (UID: \"6936032a-d78e-4d02-bb45-358368095ecf\") "
Oct 30 13:00:43.370392 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.370363    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a81a9a-e4ef-453a-8de4-96b9f403b134-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a81a9a-e4ef-453a-8de4-96b9f403b134-plugins-config-volume\") pod \"b4a81a9a-e4ef-453a-8de4-96b9f403b134\" (UID: \"b4a81a9a-e4ef-453a-8de4-96b9f403b134\") "
Oct 30 13:00:43.371081 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.371046    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6936032a-d78e-4d02-bb45-358368095ecf-plugins-config-volume" pod "6936032a-d78e-4d02-bb45-358368095ecf" (UID: "6936032a-d78e-4d02-bb45-358368095ecf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:43.371191 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.371080    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a81a9a-e4ef-453a-8de4-96b9f403b134-plugins-config-volume" pod "b4a81a9a-e4ef-453a-8de4-96b9f403b134" (UID: "b4a81a9a-e4ef-453a-8de4-96b9f403b134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:43.373459 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.373425    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6936032a-d78e-4d02-bb45-358368095ecf-kube-api-access-ltf45" pod "6936032a-d78e-4d02-bb45-358368095ecf" (UID: "6936032a-d78e-4d02-bb45-358368095ecf"). InnerVolumeSpecName "kube-api-access-ltf45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:43.373459 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.373440    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a81a9a-e4ef-453a-8de4-96b9f403b134-kube-api-access-b4jnf" pod "b4a81a9a-e4ef-453a-8de4-96b9f403b134" (UID: "b4a81a9a-e4ef-453a-8de4-96b9f403b134"). InnerVolumeSpecName "kube-api-access-b4jnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:43.472015 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.471937    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltf45\" (UniqueName: \"kubernetes.io/projected/6936032a-d78e-4d02-bb45-358368095ecf-kube-api-access-ltf45\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:43.472015 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.471964    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4jnf\" (UniqueName: \"kubernetes.io/projected/b4a81a9a-e4ef-453a-8de4-96b9f403b134-kube-api-access-b4jnf\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:43.472015 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.471973    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6936032a-d78e-4d02-bb45-358368095ecf-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:43.472015 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.471982    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a81a9a-e4ef-453a-8de4-96b9f403b134-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:43.489524 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.489490    4522 generic.go:356] "Generic (PLEG): container finished" podID="6936032a-d78e-4d02-bb45-358368095ecf" containerID="8cfc99443dc24e6ba6cebe35e523de064eea80aac582b870f1ee9573db09cdff" exitCode=0
Oct 30 13:00:43.489708 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.489545    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"
Oct 30 13:00:43.489708 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.489570    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7" event={"ID":"6936032a-d78e-4d02-bb45-358368095ecf","Type":"ContainerDied","Data":"8cfc99443dc24e6ba6cebe35e523de064eea80aac582b870f1ee9573db09cdff"}
Oct 30 13:00:43.489708 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.489628    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7" event={"ID":"6936032a-d78e-4d02-bb45-358368095ecf","Type":"ContainerDied","Data":"4e2ef238f4ad263c991fae912193fddf6fabb77ed82b2ce276c3fe386ed8f63c"}
Oct 30 13:00:43.489708 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.489655    4522 scope.go:122] "RemoveContainer" containerID="8cfc99443dc24e6ba6cebe35e523de064eea80aac582b870f1ee9573db09cdff"
Oct 30 13:00:43.490997 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.490973    4522 generic.go:356] "Generic (PLEG): container finished" podID="b4a81a9a-e4ef-453a-8de4-96b9f403b134" containerID="0cb9cab5d374374ae8a39122926680ed8441d7219d8970713c3326f92a13252c" exitCode=0
Oct 30 13:00:43.491104 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.491007    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn" event={"ID":"b4a81a9a-e4ef-453a-8de4-96b9f403b134","Type":"ContainerDied","Data":"0cb9cab5d374374ae8a39122926680ed8441d7219d8970713c3326f92a13252c"}
Oct 30 13:00:43.491104 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.491029    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"
Oct 30 13:00:43.491104 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.491039    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn" event={"ID":"b4a81a9a-e4ef-453a-8de4-96b9f403b134","Type":"ContainerDied","Data":"fbbbb5db0fd0b57a7b2162ed75a1f90924209670d6400c8f10ba6aa10422c3aa"}
Oct 30 13:00:43.504953 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.504929    4522 scope.go:122] "RemoveContainer" containerID="8cfc99443dc24e6ba6cebe35e523de064eea80aac582b870f1ee9573db09cdff"
Oct 30 13:00:43.505473 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:43.505452    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cfc99443dc24e6ba6cebe35e523de064eea80aac582b870f1ee9573db09cdff\": container with ID starting with 8cfc99443dc24e6ba6cebe35e523de064eea80aac582b870f1ee9573db09cdff not found: ID does not exist" containerID="8cfc99443dc24e6ba6cebe35e523de064eea80aac582b870f1ee9573db09cdff"
Oct 30 13:00:43.505574 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.505485    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cfc99443dc24e6ba6cebe35e523de064eea80aac582b870f1ee9573db09cdff"} err="failed to get container status \"8cfc99443dc24e6ba6cebe35e523de064eea80aac582b870f1ee9573db09cdff\": rpc error: code = NotFound desc = could not find container \"8cfc99443dc24e6ba6cebe35e523de064eea80aac582b870f1ee9573db09cdff\": container with ID starting with 8cfc99443dc24e6ba6cebe35e523de064eea80aac582b870f1ee9573db09cdff not found: ID does not exist"
Oct 30 13:00:43.505574 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.505509    4522 scope.go:122] "RemoveContainer" containerID="0cb9cab5d374374ae8a39122926680ed8441d7219d8970713c3326f92a13252c"
Oct 30 13:00:43.514416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.514398    4522 scope.go:122] "RemoveContainer" containerID="0cb9cab5d374374ae8a39122926680ed8441d7219d8970713c3326f92a13252c"
Oct 30 13:00:43.514694 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:43.514676    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb9cab5d374374ae8a39122926680ed8441d7219d8970713c3326f92a13252c\": container with ID starting with 0cb9cab5d374374ae8a39122926680ed8441d7219d8970713c3326f92a13252c not found: ID does not exist" containerID="0cb9cab5d374374ae8a39122926680ed8441d7219d8970713c3326f92a13252c"
Oct 30 13:00:43.514743 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.514701    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb9cab5d374374ae8a39122926680ed8441d7219d8970713c3326f92a13252c"} err="failed to get container status \"0cb9cab5d374374ae8a39122926680ed8441d7219d8970713c3326f92a13252c\": rpc error: code = NotFound desc = could not find container \"0cb9cab5d374374ae8a39122926680ed8441d7219d8970713c3326f92a13252c\": container with ID starting with 0cb9cab5d374374ae8a39122926680ed8441d7219d8970713c3326f92a13252c not found: ID does not exist"
Oct 30 13:00:43.541660 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.541633    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"]
Oct 30 13:00:43.545035 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.545012    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-026c2f-c8ff686cf-r47m7"]
Oct 30 13:00:43.576516 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.576493    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"]
Oct 30 13:00:43.582535 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:43.582512    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-026c2f-8fbb864b6-xrvjn"]
Oct 30 13:00:44.033650 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:44.033612    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:44.335828 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:44.335796    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6936032a-d78e-4d02-bb45-358368095ecf" path="/var/lib/kubelet/pods/6936032a-d78e-4d02-bb45-358368095ecf/volumes"
Oct 30 13:00:44.336330 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:44.336314    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a81a9a-e4ef-453a-8de4-96b9f403b134" path="/var/lib/kubelet/pods/b4a81a9a-e4ef-453a-8de4-96b9f403b134/volumes"
Oct 30 13:00:46.079651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.079622    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 13:00:46.110198 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.110166    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-socket\") pod \"d8424038-b3b1-4839-9bc2-50ee8ef95419\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") "
Oct 30 13:00:46.110406 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.110219    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-certs\") pod \"d8424038-b3b1-4839-9bc2-50ee8ef95419\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") "
Oct 30 13:00:46.110406 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.110253    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-podinfo\") pod \"d8424038-b3b1-4839-9bc2-50ee8ef95419\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") "
Oct 30 13:00:46.110406 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.110288    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8424038-b3b1-4839-9bc2-50ee8ef95419-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8424038-b3b1-4839-9bc2-50ee8ef95419-istiod-ca-cert\") pod \"d8424038-b3b1-4839-9bc2-50ee8ef95419\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") "
Oct 30 13:00:46.110583 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.110434    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-socket" pod "d8424038-b3b1-4839-9bc2-50ee8ef95419" (UID: "d8424038-b3b1-4839-9bc2-50ee8ef95419"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.110583 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.110487    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-kube-api-access-7d6p7\" (UniqueName: \"kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-kube-api-access-7d6p7\") pod \"d8424038-b3b1-4839-9bc2-50ee8ef95419\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") "
Oct 30 13:00:46.110583 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.110561    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-token\" (UniqueName: \"kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-token\") pod \"d8424038-b3b1-4839-9bc2-50ee8ef95419\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") "
Oct 30 13:00:46.110731 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.110560    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-certs" pod "d8424038-b3b1-4839-9bc2-50ee8ef95419" (UID: "d8424038-b3b1-4839-9bc2-50ee8ef95419"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.110793 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.110709    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8424038-b3b1-4839-9bc2-50ee8ef95419-istiod-ca-cert" pod "d8424038-b3b1-4839-9bc2-50ee8ef95419" (UID: "d8424038-b3b1-4839-9bc2-50ee8ef95419"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:46.110793 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.110719    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-data\") pod \"d8424038-b3b1-4839-9bc2-50ee8ef95419\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") "
Oct 30 13:00:46.110923 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.110846    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-envoy\") pod \"d8424038-b3b1-4839-9bc2-50ee8ef95419\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") "
Oct 30 13:00:46.110923 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.110907    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-credential-socket\") pod \"d8424038-b3b1-4839-9bc2-50ee8ef95419\" (UID: \"d8424038-b3b1-4839-9bc2-50ee8ef95419\") "
Oct 30 13:00:46.111025 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.110961    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-data" pod "d8424038-b3b1-4839-9bc2-50ee8ef95419" (UID: "d8424038-b3b1-4839-9bc2-50ee8ef95419"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.111261 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.111232    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-credential-socket" pod "d8424038-b3b1-4839-9bc2-50ee8ef95419" (UID: "d8424038-b3b1-4839-9bc2-50ee8ef95419"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.111830 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.111805    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.111932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.111835    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.111932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.111851    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.111932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.111866    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.111932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.111879    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8424038-b3b1-4839-9bc2-50ee8ef95419-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.112720 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.112689    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-podinfo" pod "d8424038-b3b1-4839-9bc2-50ee8ef95419" (UID: "d8424038-b3b1-4839-9bc2-50ee8ef95419"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:00:46.112837 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.112817    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-token" pod "d8424038-b3b1-4839-9bc2-50ee8ef95419" (UID: "d8424038-b3b1-4839-9bc2-50ee8ef95419"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:46.112899 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.112844    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-envoy" pod "d8424038-b3b1-4839-9bc2-50ee8ef95419" (UID: "d8424038-b3b1-4839-9bc2-50ee8ef95419"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:00:46.113001 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.112985    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-kube-api-access-7d6p7" pod "d8424038-b3b1-4839-9bc2-50ee8ef95419" (UID: "d8424038-b3b1-4839-9bc2-50ee8ef95419"). InnerVolumeSpecName "kube-api-access-7d6p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:46.213087 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.213053    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.213087 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.213084    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d6p7\" (UniqueName: \"kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-kube-api-access-7d6p7\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.213303 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.213096    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.213303 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.213104    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8424038-b3b1-4839-9bc2-50ee8ef95419-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:00:46.324025 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.323937    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:46.519476 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.519435    4522 generic.go:356] "Generic (PLEG): container finished" podID="d8424038-b3b1-4839-9bc2-50ee8ef95419" containerID="9557a40c3d0ff700121421fef27f3027a8c7fb55149cd3c287a28a457b9bfff0" exitCode=0
Oct 30 13:00:46.519725 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.519549    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc" event={"ID":"d8424038-b3b1-4839-9bc2-50ee8ef95419","Type":"ContainerDied","Data":"9557a40c3d0ff700121421fef27f3027a8c7fb55149cd3c287a28a457b9bfff0"}
Oct 30 13:00:46.519725 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.519594    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"
Oct 30 13:00:46.519725 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.519608    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc" event={"ID":"d8424038-b3b1-4839-9bc2-50ee8ef95419","Type":"ContainerDied","Data":"15c26d8ba13bc85a4af0ab212250e5172fcd8f08a910b21f4b7f86d7cc528eeb"}
Oct 30 13:00:46.519725 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.519640    4522 scope.go:122] "RemoveContainer" containerID="9557a40c3d0ff700121421fef27f3027a8c7fb55149cd3c287a28a457b9bfff0"
Oct 30 13:00:46.531223 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.531204    4522 scope.go:122] "RemoveContainer" containerID="9557a40c3d0ff700121421fef27f3027a8c7fb55149cd3c287a28a457b9bfff0"
Oct 30 13:00:46.531586 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:46.531553    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9557a40c3d0ff700121421fef27f3027a8c7fb55149cd3c287a28a457b9bfff0\": container with ID starting with 9557a40c3d0ff700121421fef27f3027a8c7fb55149cd3c287a28a457b9bfff0 not found: ID does not exist" containerID="9557a40c3d0ff700121421fef27f3027a8c7fb55149cd3c287a28a457b9bfff0"
Oct 30 13:00:46.531682 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.531594    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9557a40c3d0ff700121421fef27f3027a8c7fb55149cd3c287a28a457b9bfff0"} err="failed to get container status \"9557a40c3d0ff700121421fef27f3027a8c7fb55149cd3c287a28a457b9bfff0\": rpc error: code = NotFound desc = could not find container \"9557a40c3d0ff700121421fef27f3027a8c7fb55149cd3c287a28a457b9bfff0\": container with ID starting with 9557a40c3d0ff700121421fef27f3027a8c7fb55149cd3c287a28a457b9bfff0 not found: ID does not exist"
Oct 30 13:00:46.571035 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.571001    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"]
Oct 30 13:00:46.582873 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:46.582781    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2bfb1b-openshift-default-79d9495f97-kcbjc"]
Oct 30 13:00:48.226478 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:48.226442    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:48.326065 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:48.326033    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:48.335959 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:48.335922    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8424038-b3b1-4839-9bc2-50ee8ef95419" path="/var/lib/kubelet/pods/d8424038-b3b1-4839-9bc2-50ee8ef95419/volumes"
Oct 30 13:00:48.486916 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:48.478425    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:48.813656 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:48.813622    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:50.308207 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:50.308174    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:50.703404 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:50.703291    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:51.093524 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:51.093483    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:51.135576 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:00:51.135535    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:00:51.297972 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:51.297933    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:52.210363 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:52.210319    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:53.862901 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:53.862858    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:56.026673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:56.026635    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:57.849081 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:57.849051    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:59.031062 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:59.031019    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:59.395580 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:00:59.395436    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:00.298012 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:00.297976    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:01.773037 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:01.772995    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:01.965553 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:01.965515    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:02.066242 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:02.066150    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:02.459432 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:02.458672    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:02.562510 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:02.562472    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:02.658555 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:02.658518    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:03.071015 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:03.070974    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:04.353560 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:04.353529    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:06.143980 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:06.143919    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:08.478337 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:08.478304    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:09.114985 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:09.114954    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:09.224071 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:09.224013    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:09.405942 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:09.405812    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"]
Oct 30 13:01:09.406128 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:09.406100    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerName="istio-proxy" containerID="cri-o://eeac5e62da8ca49b63d8cacc6757c86daf70d6e6168216ea8b84f2e041afd37a" gracePeriod=30
Oct 30 13:01:09.440108 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:09.440076    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:09.765605 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:09.765426    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:09.918112 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:09.918073    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:11.016820 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:11.016775    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:11.122444 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:11.122401    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:11.434706 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:11.434656    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:11.866560 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:11.866525    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:12.936056 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:12.936025    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:14.688307 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.688231    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 13:01:14.775037 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.774989    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istiod-ca-cert\") pod \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") "
Oct 30 13:01:14.775234 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.775121    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-data\") pod \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") "
Oct 30 13:01:14.775304 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.775294    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-envoy\") pod \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") "
Oct 30 13:01:14.775384 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.775333    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-socket\") pod \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") "
Oct 30 13:01:14.775450 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.775403    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-podinfo\") pod \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") "
Oct 30 13:01:14.775507 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.775473    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-token\" (UniqueName: \"kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-token\") pod \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") "
Oct 30 13:01:14.775562 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.775517    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-data" pod "54c66945-0e5f-44d3-b5ec-228ddc468ffb" (UID: "54c66945-0e5f-44d3-b5ec-228ddc468ffb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:01:14.775562 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.775538    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-credential-socket\") pod \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") "
Oct 30 13:01:14.775682 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.775555    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istiod-ca-cert" pod "54c66945-0e5f-44d3-b5ec-228ddc468ffb" (UID: "54c66945-0e5f-44d3-b5ec-228ddc468ffb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:14.775682 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.775577    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-kube-api-access-qth92\" (UniqueName: \"kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-kube-api-access-qth92\") pod \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") "
Oct 30 13:01:14.775682 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.775669    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-socket" pod "54c66945-0e5f-44d3-b5ec-228ddc468ffb" (UID: "54c66945-0e5f-44d3-b5ec-228ddc468ffb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:01:14.775836 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.775674    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-certs\") pod \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\" (UID: \"54c66945-0e5f-44d3-b5ec-228ddc468ffb\") "
Oct 30 13:01:14.775892 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.775827    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-credential-socket" pod "54c66945-0e5f-44d3-b5ec-228ddc468ffb" (UID: "54c66945-0e5f-44d3-b5ec-228ddc468ffb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:01:14.775962 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.775903    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-certs" pod "54c66945-0e5f-44d3-b5ec-228ddc468ffb" (UID: "54c66945-0e5f-44d3-b5ec-228ddc468ffb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:01:14.776414 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.776386    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:14.776538 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.776417    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:14.776538 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.776433    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:14.776538 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.776445    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:14.776538 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.776457    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:14.777712 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.777689    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-podinfo" pod "54c66945-0e5f-44d3-b5ec-228ddc468ffb" (UID: "54c66945-0e5f-44d3-b5ec-228ddc468ffb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:01:14.778183 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.778159    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-kube-api-access-qth92" pod "54c66945-0e5f-44d3-b5ec-228ddc468ffb" (UID: "54c66945-0e5f-44d3-b5ec-228ddc468ffb"). InnerVolumeSpecName "kube-api-access-qth92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:14.778405 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.778372    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-token" pod "54c66945-0e5f-44d3-b5ec-228ddc468ffb" (UID: "54c66945-0e5f-44d3-b5ec-228ddc468ffb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:14.778500 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.778414    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-envoy" pod "54c66945-0e5f-44d3-b5ec-228ddc468ffb" (UID: "54c66945-0e5f-44d3-b5ec-228ddc468ffb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:01:14.815078 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.814620    4522 generic.go:356] "Generic (PLEG): container finished" podID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerID="eeac5e62da8ca49b63d8cacc6757c86daf70d6e6168216ea8b84f2e041afd37a" exitCode=0
Oct 30 13:01:14.815078 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.814665    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" event={"ID":"54c66945-0e5f-44d3-b5ec-228ddc468ffb","Type":"ContainerDied","Data":"eeac5e62da8ca49b63d8cacc6757c86daf70d6e6168216ea8b84f2e041afd37a"}
Oct 30 13:01:14.815078 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.814711    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"
Oct 30 13:01:14.815078 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.814737    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8" event={"ID":"54c66945-0e5f-44d3-b5ec-228ddc468ffb","Type":"ContainerDied","Data":"e767d2c1b16a4de09fe9b5ac22006d2fc62594bcb88b1036540be3b4af3d1cf1"}
Oct 30 13:01:14.815078 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.814761    4522 scope.go:122] "RemoveContainer" containerID="eeac5e62da8ca49b63d8cacc6757c86daf70d6e6168216ea8b84f2e041afd37a"
Oct 30 13:01:14.828082 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.828057    4522 scope.go:122] "RemoveContainer" containerID="fc9886805ac5845ffdd9e65fc024783e920d62af173d339af8e96bb9a8035e26"
Oct 30 13:01:14.830501 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.830476    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:14.838276 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.838253    4522 scope.go:122] "RemoveContainer" containerID="eeac5e62da8ca49b63d8cacc6757c86daf70d6e6168216ea8b84f2e041afd37a"
Oct 30 13:01:14.838624 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:01:14.838601    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeac5e62da8ca49b63d8cacc6757c86daf70d6e6168216ea8b84f2e041afd37a\": container with ID starting with eeac5e62da8ca49b63d8cacc6757c86daf70d6e6168216ea8b84f2e041afd37a not found: ID does not exist" containerID="eeac5e62da8ca49b63d8cacc6757c86daf70d6e6168216ea8b84f2e041afd37a"
Oct 30 13:01:14.838742 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.838638    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeac5e62da8ca49b63d8cacc6757c86daf70d6e6168216ea8b84f2e041afd37a"} err="failed to get container status \"eeac5e62da8ca49b63d8cacc6757c86daf70d6e6168216ea8b84f2e041afd37a\": rpc error: code = NotFound desc = could not find container \"eeac5e62da8ca49b63d8cacc6757c86daf70d6e6168216ea8b84f2e041afd37a\": container with ID starting with eeac5e62da8ca49b63d8cacc6757c86daf70d6e6168216ea8b84f2e041afd37a not found: ID does not exist"
Oct 30 13:01:14.838742 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.838663    4522 scope.go:122] "RemoveContainer" containerID="fc9886805ac5845ffdd9e65fc024783e920d62af173d339af8e96bb9a8035e26"
Oct 30 13:01:14.838975 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:01:14.838952    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9886805ac5845ffdd9e65fc024783e920d62af173d339af8e96bb9a8035e26\": container with ID starting with fc9886805ac5845ffdd9e65fc024783e920d62af173d339af8e96bb9a8035e26 not found: ID does not exist" containerID="fc9886805ac5845ffdd9e65fc024783e920d62af173d339af8e96bb9a8035e26"
Oct 30 13:01:14.839042 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.838984    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9886805ac5845ffdd9e65fc024783e920d62af173d339af8e96bb9a8035e26"} err="failed to get container status \"fc9886805ac5845ffdd9e65fc024783e920d62af173d339af8e96bb9a8035e26\": rpc error: code = NotFound desc = could not find container \"fc9886805ac5845ffdd9e65fc024783e920d62af173d339af8e96bb9a8035e26\": container with ID starting with fc9886805ac5845ffdd9e65fc024783e920d62af173d339af8e96bb9a8035e26 not found: ID does not exist"
Oct 30 13:01:14.868282 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.868246    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"]
Oct 30 13:01:14.880701 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.873103    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0a01c-openshift-default-7f5fd56c7f-86dh8"]
Oct 30 13:01:14.882106 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.882075    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qth92\" (UniqueName: \"kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-kube-api-access-qth92\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:14.882106 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.882107    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:14.882298 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.882117    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:14.882298 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:14.882125    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c66945-0e5f-44d3-b5ec-228ddc468ffb-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:15.216094 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.216063    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:15.337429 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.337381    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:15.474107 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.474003    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:15.672287 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.672237    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"]
Oct 30 13:01:15.673644 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673615    4522 cpu_manager.go:414] "containerMap: removing container" podUID="b4a81a9a-e4ef-453a-8de4-96b9f403b134" containerName="payload-processing"
Oct 30 13:01:15.673644 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673645    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a81a9a-e4ef-453a-8de4-96b9f403b134" containerName="payload-processing"
Oct 30 13:01:15.673811 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673667    4522 cpu_manager.go:414] "containerMap: removing container" podUID="6936032a-d78e-4d02-bb45-358368095ecf" containerName="payload-pre-processing"
Oct 30 13:01:15.673811 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673674    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6936032a-d78e-4d02-bb45-358368095ecf" containerName="payload-pre-processing"
Oct 30 13:01:15.673811 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673703    4522 cpu_manager.go:414] "containerMap: removing container" podUID="d8424038-b3b1-4839-9bc2-50ee8ef95419" containerName="istio-proxy"
Oct 30 13:01:15.673811 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673712    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8424038-b3b1-4839-9bc2-50ee8ef95419" containerName="istio-proxy"
Oct 30 13:01:15.673811 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673731    4522 cpu_manager.go:414] "containerMap: removing container" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerName="istio-proxy"
Oct 30 13:01:15.673811 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673739    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerName="istio-proxy"
Oct 30 13:01:15.673811 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673797    4522 cpu_manager.go:414] "containerMap: removing container" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerName="istio-proxy"
Oct 30 13:01:15.673811 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673805    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerName="istio-proxy"
Oct 30 13:01:15.674082 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673952    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerName="istio-proxy"
Oct 30 13:01:15.674082 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673966    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a81a9a-e4ef-453a-8de4-96b9f403b134" containerName="payload-processing"
Oct 30 13:01:15.674082 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673978    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6936032a-d78e-4d02-bb45-358368095ecf" containerName="payload-pre-processing"
Oct 30 13:01:15.674082 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.673991    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8424038-b3b1-4839-9bc2-50ee8ef95419" containerName="istio-proxy"
Oct 30 13:01:15.679144 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.679122    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:15.708132 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.708094    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a81a48-openshift-default-dockercfg-tz74w\""
Oct 30 13:01:15.711857 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.711827    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"]
Oct 30 13:01:15.720421 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.720387    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:15.817377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.801714    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1300d77d-620e-4675-8673-23ac3742083a-istiod-ca-cert\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:15.817377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.801837    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-envoy\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:15.817377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.801894    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjb7r\" (UniqueName: \"kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-kube-api-access-pjb7r\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:15.893379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.802277    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-istio-token\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:15.893379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.893227    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-credential-socket\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:15.893379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.893269    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-socket\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:15.893672 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.893423    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-certs\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:15.893672 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.893452    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-data\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:15.893672 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.893545    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1300d77d-620e-4675-8673-23ac3742083a-istio-podinfo\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:15.893824 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:15.847326    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:16.009375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.007442    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-certs\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.009375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.007497    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-data\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.009375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.007574    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1300d77d-620e-4675-8673-23ac3742083a-istio-podinfo\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.009375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.007648    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1300d77d-620e-4675-8673-23ac3742083a-istiod-ca-cert\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.009375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.007695    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-envoy\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.009375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.007735    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjb7r\" (UniqueName: \"kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-kube-api-access-pjb7r\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.009375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.007863    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-istio-token\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.009375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.007914    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-credential-socket\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.009375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.007947    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-socket\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.009375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.008518    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-socket\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.009375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.008760    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-certs\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.009375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.008985    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-data\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.016889 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.015614    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1300d77d-620e-4675-8673-23ac3742083a-istiod-ca-cert\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.016889 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.016510    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-credential-socket\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.044378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.043939    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1300d77d-620e-4675-8673-23ac3742083a-istio-podinfo\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.044378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.044148    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-envoy\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.062424 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.050529    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjb7r\" (UniqueName: \"kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-kube-api-access-pjb7r\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.062424 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.055335    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-istio-token\") pod \"e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") " pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.310384 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.309457    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:16.424388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.424016    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" path="/var/lib/kubelet/pods/54c66945-0e5f-44d3-b5ec-228ddc468ffb/volumes"
Oct 30 13:01:16.826630 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.826599    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"]
Oct 30 13:01:16.826990 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:01:16.826753    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1300d77d_620e_4675_8673_23ac3742083a.slice/crio-f89880eefcb5fbbfeaad6896afe8f497acf243153b422e4c1269a5e9aebc2b41 WatchSource:0}: Error finding container f89880eefcb5fbbfeaad6896afe8f497acf243153b422e4c1269a5e9aebc2b41: Status 404 returned error can't find the container with id f89880eefcb5fbbfeaad6896afe8f497acf243153b422e4c1269a5e9aebc2b41
Oct 30 13:01:16.829322 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.829285    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:01:16.829479 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.829381    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:01:16.829479 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.829429    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:01:16.903632 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.903606    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" event={"ID":"1300d77d-620e-4675-8673-23ac3742083a","Type":"ContainerStarted","Data":"f89880eefcb5fbbfeaad6896afe8f497acf243153b422e4c1269a5e9aebc2b41"}
Oct 30 13:01:16.985810 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:16.985769    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:17.112610 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.112517    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:17.234435 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.234400    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"]
Oct 30 13:01:17.235122 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.235105    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c66945-0e5f-44d3-b5ec-228ddc468ffb" containerName="istio-proxy"
Oct 30 13:01:17.237736 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.237714    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:17.238565 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.238549    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"
Oct 30 13:01:17.257979 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.257931    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-87f1b5\""
Oct 30 13:01:17.258162 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.258022    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-87f1b5-dockercfg-2nnk8\""
Oct 30 13:01:17.262099 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.262066    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"]
Oct 30 13:01:17.409490 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.405327    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqvhp\" (UniqueName: \"kubernetes.io/projected/df3a485d-9d13-4d54-b025-4ee740558a4b-kube-api-access-bqvhp\") pod \"payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w\" (UID: \"df3a485d-9d13-4d54-b025-4ee740558a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"
Oct 30 13:01:17.409490 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.405651    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3a485d-9d13-4d54-b025-4ee740558a4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w\" (UID: \"df3a485d-9d13-4d54-b025-4ee740558a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"
Oct 30 13:01:17.547812 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.508017    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3a485d-9d13-4d54-b025-4ee740558a4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w\" (UID: \"df3a485d-9d13-4d54-b025-4ee740558a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"
Oct 30 13:01:17.547812 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.508242    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqvhp\" (UniqueName: \"kubernetes.io/projected/df3a485d-9d13-4d54-b025-4ee740558a4b-kube-api-access-bqvhp\") pod \"payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w\" (UID: \"df3a485d-9d13-4d54-b025-4ee740558a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"
Oct 30 13:01:17.547812 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.509544    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3a485d-9d13-4d54-b025-4ee740558a4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w\" (UID: \"df3a485d-9d13-4d54-b025-4ee740558a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"
Oct 30 13:01:17.617064 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.613348    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqvhp\" (UniqueName: \"kubernetes.io/projected/df3a485d-9d13-4d54-b025-4ee740558a4b-kube-api-access-bqvhp\") pod \"payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w\" (UID: \"df3a485d-9d13-4d54-b025-4ee740558a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"
Oct 30 13:01:17.617755 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.617727    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"
Oct 30 13:01:17.673515 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.673013    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"]
Oct 30 13:01:17.711367 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.711316    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"]
Oct 30 13:01:17.814339 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.814303    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:17.928437 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.914047    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:17.971681 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:17.962226    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" event={"ID":"1300d77d-620e-4675-8673-23ac3742083a","Type":"ContainerStarted","Data":"a5d7baab680b3f004bd8c41d310fad53bc9259084dac28a00525fbaeaf0ce064"}
Oct 30 13:01:17.988616 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:01:17.988583    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3a485d_9d13_4d54_b025_4ee740558a4b.slice/crio-bab57b5ab758d7c0556d6209c0a5d6302fea2522c0d2399122966d727441ef3e WatchSource:0}: Error finding container bab57b5ab758d7c0556d6209c0a5d6302fea2522c0d2399122966d727441ef3e: Status 404 returned error can't find the container with id bab57b5ab758d7c0556d6209c0a5d6302fea2522c0d2399122966d727441ef3e
Oct 30 13:01:18.006545 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:18.006502    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"]
Oct 30 13:01:18.022923 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:18.018306    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" podStartSLOduration=3.018286523 podStartE2EDuration="3.018286523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:18.012778015 +0000 UTC m=+1521.828104504" watchObservedRunningTime="2026-10-30 13:01:18.018286523 +0000 UTC m=+1521.833613007"
Oct 30 13:01:18.309929 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:18.309896    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:18.315899 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:18.315877    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:18.971246 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:18.971203    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w" event={"ID":"df3a485d-9d13-4d54-b025-4ee740558a4b","Type":"ContainerStarted","Data":"7b7ff3fe9800bffd11c468989d1373fdd15a0c90063e9e3a23b793a0828bf3af"}
Oct 30 13:01:18.971773 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:18.971259    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w" event={"ID":"df3a485d-9d13-4d54-b025-4ee740558a4b","Type":"ContainerStarted","Data":"bab57b5ab758d7c0556d6209c0a5d6302fea2522c0d2399122966d727441ef3e"}
Oct 30 13:01:18.971773 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:18.971468    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:18.971773 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:18.971512    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"
Oct 30 13:01:18.972474 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:18.972457    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:01:19.011626 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:19.011576    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w" podStartSLOduration=2.011562904 podStartE2EDuration="2.011562904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:19.010204618 +0000 UTC m=+1522.825531109" watchObservedRunningTime="2026-10-30 13:01:19.011562904 +0000 UTC m=+1522.826889393"
Oct 30 13:01:19.500522 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:19.500487    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:20.496402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:20.496338    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:20.605088 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:20.605050    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:21.137484 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:01:21.137429    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:01:21.444497 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:21.444395    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:22.162111 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.162078    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:22.257994 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.257945    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:22.291051 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.291020    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"]
Oct 30 13:01:22.295321 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.295299    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"
Oct 30 13:01:22.314784 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.314755    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"]
Oct 30 13:01:22.320476 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.320446    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a81a48\""
Oct 30 13:01:22.373990 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.373957    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11aa577c-2004-4cbe-9a89-dc93c6d33053-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs\" (UID: \"11aa577c-2004-4cbe-9a89-dc93c6d33053\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"
Oct 30 13:01:22.374190 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.374022    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxd5h\" (UniqueName: \"kubernetes.io/projected/11aa577c-2004-4cbe-9a89-dc93c6d33053-kube-api-access-qxd5h\") pod \"payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs\" (UID: \"11aa577c-2004-4cbe-9a89-dc93c6d33053\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"
Oct 30 13:01:22.523383 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.520643    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11aa577c-2004-4cbe-9a89-dc93c6d33053-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs\" (UID: \"11aa577c-2004-4cbe-9a89-dc93c6d33053\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"
Oct 30 13:01:22.523383 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.520745    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxd5h\" (UniqueName: \"kubernetes.io/projected/11aa577c-2004-4cbe-9a89-dc93c6d33053-kube-api-access-qxd5h\") pod \"payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs\" (UID: \"11aa577c-2004-4cbe-9a89-dc93c6d33053\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"
Oct 30 13:01:22.523383 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.522152    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11aa577c-2004-4cbe-9a89-dc93c6d33053-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs\" (UID: \"11aa577c-2004-4cbe-9a89-dc93c6d33053\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"
Oct 30 13:01:22.584377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.581080    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxd5h\" (UniqueName: \"kubernetes.io/projected/11aa577c-2004-4cbe-9a89-dc93c6d33053-kube-api-access-qxd5h\") pod \"payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs\" (UID: \"11aa577c-2004-4cbe-9a89-dc93c6d33053\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"
Oct 30 13:01:22.617376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.616616    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"
Oct 30 13:01:22.679375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.675783    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn" podUID="19c14230-554c-403a-b08b-88db63161d71" containerName="payload-processing" containerID="cri-o://a74425809d967a093e3243170d51be940f43e0c2a2ff2f81d1ae835401196422" gracePeriod=25
Oct 30 13:01:22.712540 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:22.712479    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv" podUID="796b8883-18c7-4535-bc2a-eb8b09f78e9e" containerName="payload-pre-processing" containerID="cri-o://ae7032ae1db4ce5d0d19c2d615693c91ad39fa7286823c8f2aeb89553076ad69" gracePeriod=25
Oct 30 13:01:23.014158 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.014125    4522 generic.go:356] "Generic (PLEG): container finished" podID="19c14230-554c-403a-b08b-88db63161d71" containerID="a74425809d967a093e3243170d51be940f43e0c2a2ff2f81d1ae835401196422" exitCode=0
Oct 30 13:01:23.014376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.014217    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn" event={"ID":"19c14230-554c-403a-b08b-88db63161d71","Type":"ContainerDied","Data":"a74425809d967a093e3243170d51be940f43e0c2a2ff2f81d1ae835401196422"}
Oct 30 13:01:23.016080 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.016059    4522 generic.go:356] "Generic (PLEG): container finished" podID="796b8883-18c7-4535-bc2a-eb8b09f78e9e" containerID="ae7032ae1db4ce5d0d19c2d615693c91ad39fa7286823c8f2aeb89553076ad69" exitCode=0
Oct 30 13:01:23.016200 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.016135    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv" event={"ID":"796b8883-18c7-4535-bc2a-eb8b09f78e9e","Type":"ContainerDied","Data":"ae7032ae1db4ce5d0d19c2d615693c91ad39fa7286823c8f2aeb89553076ad69"}
Oct 30 13:01:23.037685 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.037659    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"
Oct 30 13:01:23.052462 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:01:23.052437    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11aa577c_2004_4cbe_9a89_dc93c6d33053.slice/crio-f5726d62b01424eb9696eb5f304b6c938781f14d74dcbb2130d0b2c28f2a4fc2 WatchSource:0}: Error finding container f5726d62b01424eb9696eb5f304b6c938781f14d74dcbb2130d0b2c28f2a4fc2: Status 404 returned error can't find the container with id f5726d62b01424eb9696eb5f304b6c938781f14d74dcbb2130d0b2c28f2a4fc2
Oct 30 13:01:23.053580 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.053537    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"]
Oct 30 13:01:23.067399 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.067373    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"
Oct 30 13:01:23.136530 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.136497    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c14230-554c-403a-b08b-88db63161d71-kube-api-access-lr5vp\" (UniqueName: \"kubernetes.io/projected/19c14230-554c-403a-b08b-88db63161d71-kube-api-access-lr5vp\") pod \"19c14230-554c-403a-b08b-88db63161d71\" (UID: \"19c14230-554c-403a-b08b-88db63161d71\") "
Oct 30 13:01:23.136679 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.136584    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c14230-554c-403a-b08b-88db63161d71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c14230-554c-403a-b08b-88db63161d71-plugins-config-volume\") pod \"19c14230-554c-403a-b08b-88db63161d71\" (UID: \"19c14230-554c-403a-b08b-88db63161d71\") "
Oct 30 13:01:23.137703 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.137674    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c14230-554c-403a-b08b-88db63161d71-plugins-config-volume" pod "19c14230-554c-403a-b08b-88db63161d71" (UID: "19c14230-554c-403a-b08b-88db63161d71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:23.157409 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.155828    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c14230-554c-403a-b08b-88db63161d71-kube-api-access-lr5vp" pod "19c14230-554c-403a-b08b-88db63161d71" (UID: "19c14230-554c-403a-b08b-88db63161d71"). InnerVolumeSpecName "kube-api-access-lr5vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:23.238396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.238348    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/796b8883-18c7-4535-bc2a-eb8b09f78e9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796b8883-18c7-4535-bc2a-eb8b09f78e9e-plugins-config-volume\") pod \"796b8883-18c7-4535-bc2a-eb8b09f78e9e\" (UID: \"796b8883-18c7-4535-bc2a-eb8b09f78e9e\") "
Oct 30 13:01:23.238900 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.238456    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/796b8883-18c7-4535-bc2a-eb8b09f78e9e-kube-api-access-rfll4\" (UniqueName: \"kubernetes.io/projected/796b8883-18c7-4535-bc2a-eb8b09f78e9e-kube-api-access-rfll4\") pod \"796b8883-18c7-4535-bc2a-eb8b09f78e9e\" (UID: \"796b8883-18c7-4535-bc2a-eb8b09f78e9e\") "
Oct 30 13:01:23.238900 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.238768    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796b8883-18c7-4535-bc2a-eb8b09f78e9e-plugins-config-volume" pod "796b8883-18c7-4535-bc2a-eb8b09f78e9e" (UID: "796b8883-18c7-4535-bc2a-eb8b09f78e9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:23.239044 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.239024    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr5vp\" (UniqueName: \"kubernetes.io/projected/19c14230-554c-403a-b08b-88db63161d71-kube-api-access-lr5vp\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:23.239102 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.239050    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c14230-554c-403a-b08b-88db63161d71-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:23.239102 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.239065    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796b8883-18c7-4535-bc2a-eb8b09f78e9e-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:23.240563 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.240537    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796b8883-18c7-4535-bc2a-eb8b09f78e9e-kube-api-access-rfll4" pod "796b8883-18c7-4535-bc2a-eb8b09f78e9e" (UID: "796b8883-18c7-4535-bc2a-eb8b09f78e9e"). InnerVolumeSpecName "kube-api-access-rfll4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:23.340444 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.340342    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfll4\" (UniqueName: \"kubernetes.io/projected/796b8883-18c7-4535-bc2a-eb8b09f78e9e-kube-api-access-rfll4\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:23.791793 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:23.791761    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:24.100441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.099254    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs" event={"ID":"11aa577c-2004-4cbe-9a89-dc93c6d33053","Type":"ContainerStarted","Data":"277b5dd8363f8cbefd8179d85a2bd55a79237eb49192fb5a1bf019e82eb5e8fe"}
Oct 30 13:01:24.100441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.099328    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs" event={"ID":"11aa577c-2004-4cbe-9a89-dc93c6d33053","Type":"ContainerStarted","Data":"f5726d62b01424eb9696eb5f304b6c938781f14d74dcbb2130d0b2c28f2a4fc2"}
Oct 30 13:01:24.131247 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.105437    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"
Oct 30 13:01:24.131247 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.107832    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn" event={"ID":"19c14230-554c-403a-b08b-88db63161d71","Type":"ContainerDied","Data":"ac4a2ff95de3e1f2f55b92edaf25950dff4a5c22d12fed53642ee0563dcab247"}
Oct 30 13:01:24.131247 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.107897    4522 scope.go:122] "RemoveContainer" containerID="a74425809d967a093e3243170d51be940f43e0c2a2ff2f81d1ae835401196422"
Oct 30 13:01:24.131247 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.108057    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"
Oct 30 13:01:24.131247 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.113082    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv" event={"ID":"796b8883-18c7-4535-bc2a-eb8b09f78e9e","Type":"ContainerDied","Data":"678f4397e3f265a48a3e08798f6da6d1ae73e9be9fc9d1e04463cd72e3285cf2"}
Oct 30 13:01:24.131247 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.113188    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"
Oct 30 13:01:24.156850 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.152193    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs" podStartSLOduration=2.152171743 podStartE2EDuration="2.152171743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:24.147225832 +0000 UTC m=+1527.962552323" watchObservedRunningTime="2026-10-30 13:01:24.152171743 +0000 UTC m=+1527.967498234"
Oct 30 13:01:24.158469 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.157271    4522 scope.go:122] "RemoveContainer" containerID="ae7032ae1db4ce5d0d19c2d615693c91ad39fa7286823c8f2aeb89553076ad69"
Oct 30 13:01:24.215299 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.215240    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"]
Oct 30 13:01:24.226683 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.226646    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e74c27-5c8ff8b67f-bwvkv"]
Oct 30 13:01:24.260139 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.260103    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"]
Oct 30 13:01:24.382916 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.382822    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="796b8883-18c7-4535-bc2a-eb8b09f78e9e" path="/var/lib/kubelet/pods/796b8883-18c7-4535-bc2a-eb8b09f78e9e/volumes"
Oct 30 13:01:24.383671 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:24.383645    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e74c27-b5854795d-kdgjn"]
Oct 30 13:01:25.504631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:25.504595    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:26.335196 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:26.335159    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c14230-554c-403a-b08b-88db63161d71" path="/var/lib/kubelet/pods/19c14230-554c-403a-b08b-88db63161d71/volumes"
Oct 30 13:01:27.197263 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:27.197215    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:28.976552 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:28.976501    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:29.982105 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:29.982078    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"
Oct 30 13:01:31.440252 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:31.440205    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:33.660871 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:33.660837    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:34.374676 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:34.374639    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:34.941421 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:34.941320    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:35.136430 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:35.136301    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:36.109810 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:36.109772    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:36.174964 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:36.174740    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"
Oct 30 13:01:37.767694 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:37.767656    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:37.864871 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:37.864836    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:37.964510 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:37.964475    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:39.521990 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:39.521961    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:41.035081 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:41.035052    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:41.327538 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:41.327452    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:41.421613 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:41.421561    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:41.676281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:41.676189    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:41.869883 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:41.866096    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:41.960633 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:41.960551    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:42.060864 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:42.060808    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:42.401451 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:42.401417    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:43.820716 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:43.820677    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:45.743673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:45.743637    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:48.013090 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:48.013052    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:48.120605 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:48.120563    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:48.463663 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:48.463615    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:49.110501 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:49.110454    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:49.937377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:49.937309    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:51.135537 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:01:51.135497    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:01:51.697023 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:51.696985    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:53.029036 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.029001    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"]
Oct 30 13:01:53.029490 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.029194    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq" podUID="44ef1e1c-9754-4027-a727-2fafe7aded64" containerName="maas-api" containerID="cri-o://5eff633cc069a1b4a834a3f55cc61204d195ad9f19920cf9aab4f75109e39a03" gracePeriod=30
Oct 30 13:01:53.116102 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.116068    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:53.283458 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.283366    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:53.315395 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.315331    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 13:01:53.381804 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.381769    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:53.396478 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.396449    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-metrics-certs\") pod \"44ef1e1c-9754-4027-a727-2fafe7aded64\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") "
Oct 30 13:01:53.396662 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.396645    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ef1e1c-9754-4027-a727-2fafe7aded64-kube-api-access-6f58z\" (UniqueName: \"kubernetes.io/projected/44ef1e1c-9754-4027-a727-2fafe7aded64-kube-api-access-6f58z\") pod \"44ef1e1c-9754-4027-a727-2fafe7aded64\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") "
Oct 30 13:01:53.396714 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.396681    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-maas-api-tls\") pod \"44ef1e1c-9754-4027-a727-2fafe7aded64\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") "
Oct 30 13:01:53.396769 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.396736    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ef1e1c-9754-4027-a727-2fafe7aded64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ef1e1c-9754-4027-a727-2fafe7aded64-openshift-service-ca\") pod \"44ef1e1c-9754-4027-a727-2fafe7aded64\" (UID: \"44ef1e1c-9754-4027-a727-2fafe7aded64\") "
Oct 30 13:01:53.397893 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.397148    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ef1e1c-9754-4027-a727-2fafe7aded64-openshift-service-ca" pod "44ef1e1c-9754-4027-a727-2fafe7aded64" (UID: "44ef1e1c-9754-4027-a727-2fafe7aded64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:53.397893 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.397624    4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ef1e1c-9754-4027-a727-2fafe7aded64-openshift-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:53.398955 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.398933    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ef1e1c-9754-4027-a727-2fafe7aded64-kube-api-access-6f58z" pod "44ef1e1c-9754-4027-a727-2fafe7aded64" (UID: "44ef1e1c-9754-4027-a727-2fafe7aded64"). InnerVolumeSpecName "kube-api-access-6f58z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:53.399088 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.399067    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-metrics-certs" pod "44ef1e1c-9754-4027-a727-2fafe7aded64" (UID: "44ef1e1c-9754-4027-a727-2fafe7aded64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:53.399174 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.399149    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-maas-api-tls" pod "44ef1e1c-9754-4027-a727-2fafe7aded64" (UID: "44ef1e1c-9754-4027-a727-2fafe7aded64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:53.456189 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.456158    4522 generic.go:356] "Generic (PLEG): container finished" podID="44ef1e1c-9754-4027-a727-2fafe7aded64" containerID="5eff633cc069a1b4a834a3f55cc61204d195ad9f19920cf9aab4f75109e39a03" exitCode=0
Oct 30 13:01:53.456402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.456200    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"
Oct 30 13:01:53.456402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.456241    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq" event={"ID":"44ef1e1c-9754-4027-a727-2fafe7aded64","Type":"ContainerDied","Data":"5eff633cc069a1b4a834a3f55cc61204d195ad9f19920cf9aab4f75109e39a03"}
Oct 30 13:01:53.456402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.456294    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq" event={"ID":"44ef1e1c-9754-4027-a727-2fafe7aded64","Type":"ContainerDied","Data":"b311dfcac9f414820ff5b066a4edbceef90c7acb248a468d6874f69e260d67af"}
Oct 30 13:01:53.456402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.456317    4522 scope.go:122] "RemoveContainer" containerID="5eff633cc069a1b4a834a3f55cc61204d195ad9f19920cf9aab4f75109e39a03"
Oct 30 13:01:53.467970 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.467953    4522 scope.go:122] "RemoveContainer" containerID="5eff633cc069a1b4a834a3f55cc61204d195ad9f19920cf9aab4f75109e39a03"
Oct 30 13:01:53.468287 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:01:53.468268    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eff633cc069a1b4a834a3f55cc61204d195ad9f19920cf9aab4f75109e39a03\": container with ID starting with 5eff633cc069a1b4a834a3f55cc61204d195ad9f19920cf9aab4f75109e39a03 not found: ID does not exist" containerID="5eff633cc069a1b4a834a3f55cc61204d195ad9f19920cf9aab4f75109e39a03"
Oct 30 13:01:53.468364 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.468296    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eff633cc069a1b4a834a3f55cc61204d195ad9f19920cf9aab4f75109e39a03"} err="failed to get container status \"5eff633cc069a1b4a834a3f55cc61204d195ad9f19920cf9aab4f75109e39a03\": rpc error: code = NotFound desc = could not find container \"5eff633cc069a1b4a834a3f55cc61204d195ad9f19920cf9aab4f75109e39a03\": container with ID starting with 5eff633cc069a1b4a834a3f55cc61204d195ad9f19920cf9aab4f75109e39a03 not found: ID does not exist"
Oct 30 13:01:53.498946 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.498916    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:53.498946 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.498942    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f58z\" (UniqueName: \"kubernetes.io/projected/44ef1e1c-9754-4027-a727-2fafe7aded64-kube-api-access-6f58z\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:53.499115 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.498955    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ef1e1c-9754-4027-a727-2fafe7aded64-maas-api-tls\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:01:53.507107 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.507081    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"]
Oct 30 13:01:53.514373 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.511833    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dea6ee-56c7ddfb5-krvfq"]
Oct 30 13:01:53.694923 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.694822    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:53.874177 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:53.874143    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:54.048088 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:54.048050    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:54.335694 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:54.335603    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ef1e1c-9754-4027-a727-2fafe7aded64" path="/var/lib/kubelet/pods/44ef1e1c-9754-4027-a727-2fafe7aded64/volumes"
Oct 30 13:01:54.393009 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:54.392974    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:55.655087 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:55.655056    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:56.445612 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:56.445578    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:57.223621 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:57.223584    4522 scope.go:122] "RemoveContainer" containerID="cf3bbe7d668e5b5aa27a6ce6e71f95e57da90445d306c902228010a4706882bf"
Oct 30 13:01:57.885387 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:57.885335    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 13:01:58.805477 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:01:58.805442    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:01.111530 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:01.111487    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:01.151100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:01.149419    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:01.151100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:01.149942    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:02:01.176378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:01.173153    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:01.220471 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:01.220282    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:01.319436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:01.318456    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:01.419408 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:01.418784    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:02:01.439084 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:01.439048    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:01.752439 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:01.723338    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:01.799379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:01.785212    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:01.855379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:01.828586    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:01.855722 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:01.855684    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:02.148774 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.148726    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:02:02.149959 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.149927    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:02.150227 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.150204    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:02.150434 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.150413    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:02:02.150625 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.150604    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:02.151527 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.151504    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:02:02.151717 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.151702    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:02:02.242508 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.242453    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:02.319633 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.319573    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:02.327812 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.327764    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:02:02.357527 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.357482    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:02.366491 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.366440    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:02.644428 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.561557    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:02.644428 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.562015    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:02:02.884210 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.884163    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:03.080463 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.884391    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:03.080463 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:02.988000    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:03.175433 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.150662    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:02:03.179375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.153742    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:03.263407 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.153789    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:03.263407 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.153809    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:02:03.263407 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.259138    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:02:03.263407 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.259781    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:02:03.263407 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.260177    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 13:02:03.270398 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.153921    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:02:03.491093 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.491003    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:02:03.578378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.553583    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:02:03.597885 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.567024    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:02:03.598156 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.572489    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:02:03.772441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.756597    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:02:03.888301 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.888257    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:02:03.969563 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.913579    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:03.981417 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.973469    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:02:03.998678 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.998632    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:02:03.999498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:03.999472    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:02:04.139590 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:04.139483    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:02:04.246378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:04.218525    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:04.345376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:04.254892    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:04.345376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:04.255312    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:02:04.345376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:04.255332    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:04.559402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:04.551594    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:04.572532 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:04.572268    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:04.767178 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:04.767133    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:02:04.787454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:04.787396    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:04.826138 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:04.825812    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:05.069513 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.038130    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:02:05.069513 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.041296    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:02:05.069513 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.043733    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Oct 30 13:02:05.069513 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.043943    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:05.153567 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.147791    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:02:05.311615 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.311574    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:05.317390 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.312228    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:05.404440 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.339205    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:02:05.405056 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.341744    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:05.405750 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.341776    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:02:05.406109 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.341796    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:02:05.469384 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.457456    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:05.469384 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.457530    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:05.714582 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.714472    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:02:05.740530 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.739764    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:05.770391 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.769637    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:05.799524 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:05.799465    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:06.030712 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.030666    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:06.031049 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.031017    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:06.040315 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.039494    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:02:06.043850 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.043440    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:06.044155 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.044121    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:02:06.150285 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.150238    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:06.261956 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.261495    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:06.261956 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.261559    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:06.433434 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.423502    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:06.433434 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.423892    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:02:06.433434 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.424169    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:06.433434 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.424460    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:02:06.714009 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.713898    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:06.736500 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.736445    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:06.808394 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.788926    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:06.886843 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.817836    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:06.887774 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:06.818459    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:07.052562 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.052508    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:07.086380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.053660    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:07.177677 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.177626    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:07.178646 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.113556    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:07.183422 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.115683    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:07.191383 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.168106    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:02:07.407437 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.379318    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:07.433378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.430478    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:07.433378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.430927    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:02:07.433378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.431236    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:02:07.433378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.431515    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:07.681551 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.681446    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:07.967443 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.967010    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:02:07.967630 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.967438    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:02:07.983805 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:07.967748    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:08.056379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:08.055445    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:02:08.184531 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:08.184480    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:08.186517 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:08.184790    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:08.219500 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:08.194509    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:08.729621 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:08.729549    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:08.797381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:08.766546    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:08.797381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:08.766930    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:02:08.797381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:08.767205    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:02:08.797381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:08.767463    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:08.997436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:08.972056    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:08.999730 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:08.991002    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:08.999730 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:08.991056    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:02:09.100526 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:09.077628    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:09.201379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:09.193679    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:09.209382 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:09.199463    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:09.257528 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:09.252769    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:09.610845 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:09.609097    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:09.795378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:09.770739    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:09.795378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:09.771182    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:02:09.795378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:09.792266    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:02:09.795378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:09.792723    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:02:09.912772 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:09.912669    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:09.969430 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:09.969376    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:10.001650 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.001603    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:02:10.119400 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.006000    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:02:10.119400 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.006225    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:10.123378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.119782    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:10.218305 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.215460    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:10.247379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.233226    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:10.325396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.323238    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:02:10.493629 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.493517    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded"
Oct 30 13:02:10.744433 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.742278    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:10.772530 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.772480    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:10.773043 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.773013    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:10.783561 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.783516    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:10.783896 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.783855    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:10.793662 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.793600    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:10.794089 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:10.794064    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:02:11.018452 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.017448    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:02:11.021521 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.021475    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:02:11.022081 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.022045    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:11.206098 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.206037    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:11.295441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.263509    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:11.295967 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.268594    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:02:11.323094 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.272440    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:02:11.323713 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.286810    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:11.324218 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.324184    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:11.539376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.533257    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:11.778979 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.777109    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:11.778979 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.778125    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:02:11.797245 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.795450    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:11.797245 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:11.796439    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:12.022868 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.021447    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:02:12.022868 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.022070    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:12.022868 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.022490    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:12.215531 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.214476    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:12.296409 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.296341    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:12.336656 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.334459    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:12.336656 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.334858    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:02:12.514624 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.514534    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:12.601380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.536489    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:12.620667 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.620598    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:12.779672 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.779609    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:12.780409 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.780321    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:12.799658 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.799607    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:12.833705 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.804754    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:02:12.909320 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:12.909282    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:02:13.027376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:13.026729    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:13.027376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:13.027116    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:02:13.027950 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:13.027484    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:02:13.079431 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:13.075814    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:13.329557 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:13.304743    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:02:13.348381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:13.344597    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:13.362069 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:13.349011    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:02:13.819449 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:13.787542    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:13.819449 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:13.788126    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:02:13.843569 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:13.843519    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:13.844373 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:13.844320    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:13.945698 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:13.945658    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:02:14.030823 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.029332    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:14.030823 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.029662    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:02:14.030823 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.029962    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:02:14.051529 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.051487    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:14.224490 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.224373    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:14.311938 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.310893    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:14.350496 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.350450    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:14.636397 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.636338    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:14.802804 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.802746    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:14.842379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.803071    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:14.842995 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.833026    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:14.845452 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.843963    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:14.845452 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.844714    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:14.845917 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.845888    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:14.990430 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:14.980956    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:15.059381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:15.030527    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:15.059381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:15.030946    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:02:15.059381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:15.032450    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:15.082625 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:15.082574    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:15.083971 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:15.083944    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:02:15.194436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:15.176593    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:02:15.350380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:15.325868    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:15.384379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:15.355602    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:15.805517 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:15.805456    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:15.844290 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:15.843719    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:15.847755 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:15.844770    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:15.847755 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:15.845876    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:15.847755 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:15.846831    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:02:16.033587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:16.033518    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:02:16.034333 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:16.034288    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:02:16.042442 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:16.039930    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:02:16.093432 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:16.086570    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:16.201572 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:16.201525    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:16.202291 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:16.202264    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:02:16.584375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:16.580403    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:16.633476 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:16.633425    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"]
Oct 30 13:02:16.643398 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:16.642690    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:02:17.049392 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:17.034947    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:17.049886 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:17.049834    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:17.139376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:17.137455    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:02:17.139376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:17.137804    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:02:17.139376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:17.138440    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:17.257409 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:17.245646    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:17.419230 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:17.414580    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:02:17.447379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:17.427882    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:17.590387 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:17.587936    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:17.621385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:17.616980    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:02:17.649698 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:17.649653    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:17.856388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:17.846181    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:18.235577 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.235479    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:18.259415 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.236473    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 13:02:18.260472 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.249513    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:18.260910 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.249581    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:02:18.284570 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.249606    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded"
Oct 30 13:02:18.285026 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.249933    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:02:18.292374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.249958    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:02:18.292374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.249979    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:02:18.292374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.250010    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:02:18.292374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.289441    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:18.305057 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.305024    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:18.473409 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.468825    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:18.559039 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.559004    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:02:18.595734 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:18.593952    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:19.035181 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.035147    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:02:19.085463 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.081622    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:02:19.153087 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.150634    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:02:19.163828 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.162207    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:02:19.187376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.170860    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 13:02:19.187376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.180348    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:02:19.210373 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.205988    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:02:19.214388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.214221    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:02:19.241106 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.241033    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:02:19.258930 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.242288    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:19.296178 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.296131    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:19.296684 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.296642    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:02:19.325062 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.325036    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:02:19.362726 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.362692    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:19.477843 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:19.473121    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:20.045562 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:20.045530    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:20.045762 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:20.045588    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:21.030870 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:21.030836    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:21.151610 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:02:21.151562    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:02:22.141773 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.141723    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs" podUID="11aa577c-2004-4cbe-9a89-dc93c6d33053" containerName="payload-processing" containerID="cri-o://277b5dd8363f8cbefd8179d85a2bd55a79237eb49192fb5a1bf019e82eb5e8fe" gracePeriod=25
Oct 30 13:02:22.577872 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.577552    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"
Oct 30 13:02:22.589896 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.589834    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:22.670663 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.669528    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11aa577c-2004-4cbe-9a89-dc93c6d33053-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11aa577c-2004-4cbe-9a89-dc93c6d33053-plugins-config-volume\") pod \"11aa577c-2004-4cbe-9a89-dc93c6d33053\" (UID: \"11aa577c-2004-4cbe-9a89-dc93c6d33053\") "
Oct 30 13:02:22.670663 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.669636    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11aa577c-2004-4cbe-9a89-dc93c6d33053-kube-api-access-qxd5h\" (UniqueName: \"kubernetes.io/projected/11aa577c-2004-4cbe-9a89-dc93c6d33053-kube-api-access-qxd5h\") pod \"11aa577c-2004-4cbe-9a89-dc93c6d33053\" (UID: \"11aa577c-2004-4cbe-9a89-dc93c6d33053\") "
Oct 30 13:02:22.671296 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.671273    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11aa577c-2004-4cbe-9a89-dc93c6d33053-plugins-config-volume" pod "11aa577c-2004-4cbe-9a89-dc93c6d33053" (UID: "11aa577c-2004-4cbe-9a89-dc93c6d33053"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:22.674017 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.673978    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11aa577c-2004-4cbe-9a89-dc93c6d33053-kube-api-access-qxd5h" pod "11aa577c-2004-4cbe-9a89-dc93c6d33053" (UID: "11aa577c-2004-4cbe-9a89-dc93c6d33053"). InnerVolumeSpecName "kube-api-access-qxd5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:22.772698 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.772655    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11aa577c-2004-4cbe-9a89-dc93c6d33053-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:22.772698 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.772693    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxd5h\" (UniqueName: \"kubernetes.io/projected/11aa577c-2004-4cbe-9a89-dc93c6d33053-kube-api-access-qxd5h\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:22.805362 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.805215    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:22.906754 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.906701    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:22.928739 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.928709    4522 generic.go:356] "Generic (PLEG): container finished" podID="11aa577c-2004-4cbe-9a89-dc93c6d33053" containerID="277b5dd8363f8cbefd8179d85a2bd55a79237eb49192fb5a1bf019e82eb5e8fe" exitCode=0
Oct 30 13:02:22.928903 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.928777    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"
Oct 30 13:02:22.928903 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.928775    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs" event={"ID":"11aa577c-2004-4cbe-9a89-dc93c6d33053","Type":"ContainerDied","Data":"277b5dd8363f8cbefd8179d85a2bd55a79237eb49192fb5a1bf019e82eb5e8fe"}
Oct 30 13:02:22.929006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.928893    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs" event={"ID":"11aa577c-2004-4cbe-9a89-dc93c6d33053","Type":"ContainerDied","Data":"f5726d62b01424eb9696eb5f304b6c938781f14d74dcbb2130d0b2c28f2a4fc2"}
Oct 30 13:02:22.929006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.928940    4522 scope.go:122] "RemoveContainer" containerID="277b5dd8363f8cbefd8179d85a2bd55a79237eb49192fb5a1bf019e82eb5e8fe"
Oct 30 13:02:22.963468 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.963438    4522 scope.go:122] "RemoveContainer" containerID="277b5dd8363f8cbefd8179d85a2bd55a79237eb49192fb5a1bf019e82eb5e8fe"
Oct 30 13:02:22.964094 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:02:22.964056    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277b5dd8363f8cbefd8179d85a2bd55a79237eb49192fb5a1bf019e82eb5e8fe\": container with ID starting with 277b5dd8363f8cbefd8179d85a2bd55a79237eb49192fb5a1bf019e82eb5e8fe not found: ID does not exist" containerID="277b5dd8363f8cbefd8179d85a2bd55a79237eb49192fb5a1bf019e82eb5e8fe"
Oct 30 13:02:22.964249 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:22.964223    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277b5dd8363f8cbefd8179d85a2bd55a79237eb49192fb5a1bf019e82eb5e8fe"} err="failed to get container status \"277b5dd8363f8cbefd8179d85a2bd55a79237eb49192fb5a1bf019e82eb5e8fe\": rpc error: code = NotFound desc = could not find container \"277b5dd8363f8cbefd8179d85a2bd55a79237eb49192fb5a1bf019e82eb5e8fe\": container with ID starting with 277b5dd8363f8cbefd8179d85a2bd55a79237eb49192fb5a1bf019e82eb5e8fe not found: ID does not exist"
Oct 30 13:02:23.002274 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:23.002231    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"]
Oct 30 13:02:23.006634 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:23.006589    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a81a48-67cb59c596-ts5cs"]
Oct 30 13:02:23.126428 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:23.117891    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:23.480453 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:23.477694    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:23.588460 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:23.587386    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:23.731734 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:23.731603    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:24.107296 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:24.107253    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:24.107544 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:24.107327    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql" podUID="b7695fab-0b10-4ffe-857e-32411056876c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:24.179373 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:24.179322    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:24.339528 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:24.339492    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11aa577c-2004-4cbe-9a89-dc93c6d33053" path="/var/lib/kubelet/pods/11aa577c-2004-4cbe-9a89-dc93c6d33053/volumes"
Oct 30 13:02:24.525645 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:24.525609    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:24.726618 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:24.723012    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:24.854563 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:24.828547    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:25.172719 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:25.060427    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:25.172719 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:25.060471    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:25.172719 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:25.117775    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:25.362154 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:25.275900    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:26.401094 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:26.401062    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": read tcp 10.133.0.2:41810->10.133.0.111:15021: read: connection reset by peer" start-of-body=
Oct 30 13:02:26.401554 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:26.401137    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": read tcp 10.133.0.2:41810->10.133.0.111:15021: read: connection reset by peer"
Oct 30 13:02:26.827062 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:26.827023    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:27.022765 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.022724    4522 generic.go:356] "Generic (PLEG): container finished" podID="1300d77d-620e-4675-8673-23ac3742083a" containerID="a5d7baab680b3f004bd8c41d310fad53bc9259084dac28a00525fbaeaf0ce064" exitCode=0
Oct 30 13:02:27.023231 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.023199    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" event={"ID":"1300d77d-620e-4675-8673-23ac3742083a","Type":"ContainerDied","Data":"a5d7baab680b3f004bd8c41d310fad53bc9259084dac28a00525fbaeaf0ce064"}
Oct 30 13:02:27.024864 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.024840    4522 scope.go:122] "RemoveContainer" containerID="a5d7baab680b3f004bd8c41d310fad53bc9259084dac28a00525fbaeaf0ce064"
Oct 30 13:02:27.026979 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.026854    4522 generic.go:356] "Generic (PLEG): container finished" podID="74749d96-9482-4a69-90b3-9be8c58faa09" containerID="e465083f88d26f5a78107c7f001cd413bc0eb2fd7f039b92727fce371e7c8ad5" exitCode=0
Oct 30 13:02:27.026979 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.026937    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" event={"ID":"74749d96-9482-4a69-90b3-9be8c58faa09","Type":"ContainerDied","Data":"e465083f88d26f5a78107c7f001cd413bc0eb2fd7f039b92727fce371e7c8ad5"}
Oct 30 13:02:27.028596 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.027440    4522 scope.go:122] "RemoveContainer" containerID="e465083f88d26f5a78107c7f001cd413bc0eb2fd7f039b92727fce371e7c8ad5"
Oct 30 13:02:27.028596 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.027501    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:27.028596 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.027596    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:27.028596 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.027644    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:27.031280 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.029554    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:27.031280 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.029639    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:27.031280 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.029690    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:27.035375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.035327    4522 generic.go:356] "Generic (PLEG): container finished" podID="b7695fab-0b10-4ffe-857e-32411056876c" containerID="764a1ce65980d1d46d93c0561626b27d0275e415756884e64775d84280c35fba" exitCode=0
Oct 30 13:02:27.035680 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.035657    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql" event={"ID":"b7695fab-0b10-4ffe-857e-32411056876c","Type":"ContainerDied","Data":"764a1ce65980d1d46d93c0561626b27d0275e415756884e64775d84280c35fba"}
Oct 30 13:02:27.037583 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.037557    4522 scope.go:122] "RemoveContainer" containerID="764a1ce65980d1d46d93c0561626b27d0275e415756884e64775d84280c35fba"
Oct 30 13:02:27.041800 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.040075    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:27.041800 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.040248    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:27.041800 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.040299    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:27.043649 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.043619    4522 generic.go:356] "Generic (PLEG): container finished" podID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerID="5aa13ef721e419998aaf634c1f9571846d15e9d37bbd5fdb8598c877335f43ed" exitCode=0
Oct 30 13:02:27.043779 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.043760    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" event={"ID":"a8cf4110-29ff-414d-87af-dfa343a91bc7","Type":"ContainerDied","Data":"5aa13ef721e419998aaf634c1f9571846d15e9d37bbd5fdb8598c877335f43ed"}
Oct 30 13:02:27.044258 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.044211    4522 scope.go:122] "RemoveContainer" containerID="5aa13ef721e419998aaf634c1f9571846d15e9d37bbd5fdb8598c877335f43ed"
Oct 30 13:02:27.050260 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.050228    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:27.050490 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.050450    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:27.050615 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:27.050514    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:28.067548 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.067508    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" event={"ID":"1300d77d-620e-4675-8673-23ac3742083a","Type":"ContainerStarted","Data":"b69004c6c828972b1726a0e68680172c81bb34690b53669ca4d60816feffeb6d"}
Oct 30 13:02:28.071610 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.071580    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" event={"ID":"74749d96-9482-4a69-90b3-9be8c58faa09","Type":"ContainerStarted","Data":"1b9e6ad45f834a16cdfb3db99c7037a011b1c837cfbb4653d0bb96be55b12cbc"}
Oct 30 13:02:28.079145 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.079109    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql" event={"ID":"b7695fab-0b10-4ffe-857e-32411056876c","Type":"ContainerStarted","Data":"1ed3d64c4cb3a25b24be5f5c83d5b35d72ac6ed7e4bf27a61fd3edaa285903da"}
Oct 30 13:02:28.083645 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.083614    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" event={"ID":"a8cf4110-29ff-414d-87af-dfa343a91bc7","Type":"ContainerStarted","Data":"7dd1d9affb8f4e5a3e94c118c07b17cf2ae2575d91af32d21b99ba66fb3150b8"}
Oct 30 13:02:28.092767 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.092740    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 13:02:28.093082 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.093060    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 13:02:28.093227 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.093204    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql" podUID="b7695fab-0b10-4ffe-857e-32411056876c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 13:02:28.170870 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.170835    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 13:02:28.171252 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.171215    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 13:02:28.171648 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.171586    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:02:28.314768 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.314732    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:02:28.320727 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.318463    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:02:28.320727 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.318524    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:02:28.778380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.778325    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:29.001376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.990778    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 13:02:29.001376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.990954    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:02:29.001376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:28.991004    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:02:29.118437 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.118017    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:29.138376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.122486    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 13:02:29.138376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.122559    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql" podUID="b7695fab-0b10-4ffe-857e-32411056876c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 13:02:29.138376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.132588    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:29.192380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.172304    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 13:02:29.192380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.172384    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 13:02:29.272379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.251431    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:29.272379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.267857    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:29.351378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.350527    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:29.427456 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.426643    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:29.493376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.492961    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:02:29.493376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.493219    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:02:29.517376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.517273    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:29.585378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.584635    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:29.635376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:29.634710    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:30.121378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.109801    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:02:30.176379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.112452    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 13:02:30.180375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.176680    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql" podUID="b7695fab-0b10-4ffe-857e-32411056876c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 13:02:30.257378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.191485    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:02:30.257378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.252424    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:02:30.257378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.191522    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 13:02:30.257378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.253421    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 13:02:30.257378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.255857    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:30.257378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.256219    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:02:30.271564 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.271527    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:30.272079 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.272056    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:02:30.325375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.325230    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:02:30.325375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.325286    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:02:30.369704 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.369619    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:02:30.404514 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.404435    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:02:30.410163 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.410127    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:30.444669 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.444623    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:30.445136 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.445115    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 13:02:30.547388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.542345    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:30.612644 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.611265    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:30.651418 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.638380    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:30.790399 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.788892    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:02:30.790399 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.788963    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:02:30.803547 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.803500    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:30.917395 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:30.916075    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:31.003713 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.003584    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:02:31.004265 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.004147    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:02:31.062019 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.061721    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:31.352425 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.348521    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:02:31.476401 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.396481    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:02:31.481384 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.477914    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:02:31.481384 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.418693    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:02:31.490381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.420245    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:02:31.615422 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.420314    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:02:31.616550 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.420343    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:02:31.862695 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.420439    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 13:02:31.862996 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.862964    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql" podUID="b7695fab-0b10-4ffe-857e-32411056876c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 13:02:31.863119 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.500875    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:31.868446 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.499574    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:31.872371 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.583718    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:31.898194 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.423177    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:02:31.898382 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.898237    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:02:31.898382 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.872910    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:02:31.899589 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.420293    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 30 13:02:31.899589 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.424843    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 13:02:31.899765 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.899586    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 13:02:31.927377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.902079    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:31.927923 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.912929    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:31.987380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.964573    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:31.987380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.978433    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:32.008377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:31.988818    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:02:32.009471 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.009436    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:02:32.026440 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.026411    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"]
Oct 30 13:02:32.027029 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.026987    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql" podUID="b7695fab-0b10-4ffe-857e-32411056876c" containerName="istio-proxy" containerID="cri-o://1ed3d64c4cb3a25b24be5f5c83d5b35d72ac6ed7e4bf27a61fd3edaa285903da" gracePeriod=30
Oct 30 13:02:32.192601 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.192501    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 13:02:32.237309 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.237272    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 13:02:32.237925 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.236376    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:32.384509 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.384471    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:02:32.385124 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.385084    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:02:32.506542 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.506443    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:02:32.507113 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.507082    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:02:32.507515 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.507494    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:02:32.540546 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.540498    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:32.610389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.603489    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:32.626826 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.621449    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:32.739661 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.739606    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:32.899385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.870967    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:32.899385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.872111    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:32.932508 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.907291    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:32.932508 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.907424    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:32.932508 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.929474    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:32.932508 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.929947    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:02:32.969486 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.969440    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:02:32.995366 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.995314    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:02:32.995660 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.995616    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:02:32.998227 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.997938    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:02:32.998489 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:32.998474    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:02:33.054076 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.054035    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:33.136734 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.136387    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:02:33.188437 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.182738    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 13:02:33.337378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.228065    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 13:02:33.420403 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.413099    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:33.470829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.470617    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:02:33.471215 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.471084    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:02:33.567484 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.508158    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:33.567484 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.508742    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:02:33.567484 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.509003    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:02:33.567484 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.541822    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:33.680336 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.659093    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:33.692401 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.688432    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"]
Oct 30 13:02:33.692401 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.688679    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy" containerID="cri-o://7dd1d9affb8f4e5a3e94c118c07b17cf2ae2575d91af32d21b99ba66fb3150b8" gracePeriod=30
Oct 30 13:02:33.831742 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.831704    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:33.881742 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.881687    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:33.882400 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.882343    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:02:33.932614 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.932513    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:33.933342 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.933295    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:02:33.958475 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.955472    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:33.980477 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.959291    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:02:33.980477 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.978425    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:02:33.986716 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.986684    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:02:33.986966 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:33.986938    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:02:34.026517 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.026473    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:34.139585 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.136474    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:34.366376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.357517    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:02:34.366376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.357584    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:02:34.509498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.509447    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:34.510407 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.510379    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:02:34.510851 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.510829    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:02:34.685552 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.685440    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:34.804628 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.804578    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:34.892506 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.892461    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:34.893125 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.892758    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:02:34.953625 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.953521    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:34.955882 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.955847    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:34.963570 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.963522    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:34.985199 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:34.985158    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:35.003679 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.003645    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:02:35.003919 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.003893    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:02:35.029471 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.029430    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:35.112661 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.112628    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:02:35.159987 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.152990    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:02:35.159987 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.153420    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:02:35.161491 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.161463    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:35.400189 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.395825    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:02:35.416191 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.416149    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:02:35.519416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.512515    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:35.534703 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.524019    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:02:35.553681 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.524060    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:02:35.691343 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.691237    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:35.743532 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.743490    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:35.815605 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.815559    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:02:35.876144 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.876112    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:35.896377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.893894    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:35.897223 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.897191    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:35.974430 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:35.969719    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:02:36.015515 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.015477    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:02:36.015758 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.015729    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:02:36.019375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.016470    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:36.122377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.107460    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:36.184081 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.184041    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:36.343581 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.343545    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:02:36.343817 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.343791    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:02:36.523414 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.523341    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:36.571609 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.570615    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:36.764379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.761230    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:36.851379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.847397    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:36.925410 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.918003    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:36.926471 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.920132    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:02:36.962392 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.950497    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:36.962392 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:36.959525    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:37.026378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.017022    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:02:37.026378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.023710    4522 patch_prober.go:29] interesting pod/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:02:37.026378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.023775    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:02:37.051825 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.051783    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"]
Oct 30 13:02:37.075398 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.065077    4522 cpu_manager.go:414] "containerMap: removing container" podUID="19c14230-554c-403a-b08b-88db63161d71" containerName="payload-processing"
Oct 30 13:02:37.091944 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.091891    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c14230-554c-403a-b08b-88db63161d71" containerName="payload-processing"
Oct 30 13:02:37.106804 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.106756    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:37.111850 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.111761    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:02:37.176996 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.112387    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:37.177820 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.120433    4522 cpu_manager.go:414] "containerMap: removing container" podUID="796b8883-18c7-4535-bc2a-eb8b09f78e9e" containerName="payload-pre-processing"
Oct 30 13:02:37.182263 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.182236    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="796b8883-18c7-4535-bc2a-eb8b09f78e9e" containerName="payload-pre-processing"
Oct 30 13:02:37.182499 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.182480    4522 cpu_manager.go:414] "containerMap: removing container" podUID="11aa577c-2004-4cbe-9a89-dc93c6d33053" containerName="payload-processing"
Oct 30 13:02:37.186427 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.186401    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11aa577c-2004-4cbe-9a89-dc93c6d33053" containerName="payload-processing"
Oct 30 13:02:37.186624 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.186608    4522 cpu_manager.go:414] "containerMap: removing container" podUID="44ef1e1c-9754-4027-a727-2fafe7aded64" containerName="maas-api"
Oct 30 13:02:37.186753 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.186739    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ef1e1c-9754-4027-a727-2fafe7aded64" containerName="maas-api"
Oct 30 13:02:37.206376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.187368    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c14230-554c-403a-b08b-88db63161d71" containerName="payload-processing"
Oct 30 13:02:37.224378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.213434    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="796b8883-18c7-4535-bc2a-eb8b09f78e9e" containerName="payload-pre-processing"
Oct 30 13:02:37.224378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.213485    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="11aa577c-2004-4cbe-9a89-dc93c6d33053" containerName="payload-processing"
Oct 30 13:02:37.224378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.213506    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ef1e1c-9754-4027-a727-2fafe7aded64" containerName="maas-api"
Oct 30 13:02:37.225479 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.225442    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:02:37.264585 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.264545    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"]
Oct 30 13:02:37.270776 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.270692    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.300009 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.299047    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:02:37.314190 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.313501    4522 patch_prober.go:29] interesting pod/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:02:37.314190 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.313555    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:02:37.320884 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.315855    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4cb0fe-openshift-default-dockercfg-4ftfs\""
Oct 30 13:02:37.324482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.323705    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-data\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.324482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.323774    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-podinfo\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.324482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.323801    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istiod-ca-cert\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.324482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.323842    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-socket\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.324482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.323902    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twbtt\" (UniqueName: \"kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-kube-api-access-twbtt\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.324482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.323952    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-credential-socket\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.324482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.323975    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-token\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.324482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.324046    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-envoy\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.324482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.324228    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-certs\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.325375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.325268    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:02:37.331700 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.331681    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:02:37.371800 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.371773    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:02:37.409562 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.408715    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:02:37.415623 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.415600    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 13:02:37.416918 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.416900    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:02:37.431059 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.430983    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-certs\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.431370 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.431331    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-data\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.431550 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.431534    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-podinfo\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.431705 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.431688    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istiod-ca-cert\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.431826 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.431813    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-socket\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.431962 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.431949    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twbtt\" (UniqueName: \"kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-kube-api-access-twbtt\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.432093 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.432080    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-credential-socket\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.432191 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.432177    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-token\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.432392 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.432378    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-envoy\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.436113 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.436085    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-data\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.437613 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.437468    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-certs\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.438661 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.438621    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-credential-socket\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.445629 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.444989    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istiod-ca-cert\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.445788 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.445199    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-socket\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.515402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.515343    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:02:37.518125 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.518070    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twbtt\" (UniqueName: \"kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-kube-api-access-twbtt\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.518934 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.518881    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-podinfo\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.519407 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.519344    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-envoy\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.522943 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.522918    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-token\") pod \"e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") " pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.637861 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.637825    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:37.799665 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.799569    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:37.899032 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:37.898995    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:38.026376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.022049    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:38.044517 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.042926    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 13:02:38.044888 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.044833    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 13:02:38.059810 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.059703    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 13:02:38.063089 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.063000    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 13:02:38.118201 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.116584    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"]
Oct 30 13:02:38.143209 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.143153    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:38.143418 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.143274    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:38.143418 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.143319    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:38.315297 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.315215    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:02:38.316323 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.316301    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:02:38.339613 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.339580    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" event={"ID":"649cac72-df6c-4572-8fc9-a3e78a5e20b8","Type":"ContainerStarted","Data":"a88c70190f694f652b69b8a9cd45b6c6843b3520f6166ac501897ce398e7c28c"}
Oct 30 13:02:38.339838 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.339820    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" event={"ID":"649cac72-df6c-4572-8fc9-a3e78a5e20b8","Type":"ContainerStarted","Data":"36e51498f1df149d42db91110b32f0b76e8f82172b3fcdb81a539a4511612bce"}
Oct 30 13:02:38.442919 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.442855    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" podStartSLOduration=2.442836123 podStartE2EDuration="2.442836123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:38.441252233 +0000 UTC m=+1602.256578724" watchObservedRunningTime="2026-10-30 13:02:38.442836123 +0000 UTC m=+1602.258162615"
Oct 30 13:02:38.647797 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.647690    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:38.815306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.815271    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:38.919510 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:38.919443    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:39.019595 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:39.019560    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:02:39.647067 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:39.647028    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:39.837342 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:39.837305    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:39.986997 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:39.986909    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:40.469397 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:40.459552    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:40.472034 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:40.472003    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:02:40.956871 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:40.956838    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:41.074167 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:41.074133    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:42.025282 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.025217    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:42.571445 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.570998    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 13:02:42.598438 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.584493    4522 generic.go:356] "Generic (PLEG): container finished" podID="b7695fab-0b10-4ffe-857e-32411056876c" containerID="1ed3d64c4cb3a25b24be5f5c83d5b35d72ac6ed7e4bf27a61fd3edaa285903da" exitCode=0
Oct 30 13:02:42.598438 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.589730    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"
Oct 30 13:02:42.598438 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.589952    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql" event={"ID":"b7695fab-0b10-4ffe-857e-32411056876c","Type":"ContainerDied","Data":"1ed3d64c4cb3a25b24be5f5c83d5b35d72ac6ed7e4bf27a61fd3edaa285903da"}
Oct 30 13:02:42.598438 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.590012    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql" event={"ID":"b7695fab-0b10-4ffe-857e-32411056876c","Type":"ContainerDied","Data":"fa7d13347b64b9d96a7fe1d28e5c8cab28cb2389bd8cb951410608baf8fbde2f"}
Oct 30 13:02:42.598438 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.592049    4522 scope.go:122] "RemoveContainer" containerID="1ed3d64c4cb3a25b24be5f5c83d5b35d72ac6ed7e4bf27a61fd3edaa285903da"
Oct 30 13:02:42.648376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.648131    4522 scope.go:122] "RemoveContainer" containerID="764a1ce65980d1d46d93c0561626b27d0275e415756884e64775d84280c35fba"
Oct 30 13:02:42.664751 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.664732    4522 scope.go:122] "RemoveContainer" containerID="1ed3d64c4cb3a25b24be5f5c83d5b35d72ac6ed7e4bf27a61fd3edaa285903da"
Oct 30 13:02:42.665120 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:02:42.665092    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed3d64c4cb3a25b24be5f5c83d5b35d72ac6ed7e4bf27a61fd3edaa285903da\": container with ID starting with 1ed3d64c4cb3a25b24be5f5c83d5b35d72ac6ed7e4bf27a61fd3edaa285903da not found: ID does not exist" containerID="1ed3d64c4cb3a25b24be5f5c83d5b35d72ac6ed7e4bf27a61fd3edaa285903da"
Oct 30 13:02:42.665243 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.665128    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed3d64c4cb3a25b24be5f5c83d5b35d72ac6ed7e4bf27a61fd3edaa285903da"} err="failed to get container status \"1ed3d64c4cb3a25b24be5f5c83d5b35d72ac6ed7e4bf27a61fd3edaa285903da\": rpc error: code = NotFound desc = could not find container \"1ed3d64c4cb3a25b24be5f5c83d5b35d72ac6ed7e4bf27a61fd3edaa285903da\": container with ID starting with 1ed3d64c4cb3a25b24be5f5c83d5b35d72ac6ed7e4bf27a61fd3edaa285903da not found: ID does not exist"
Oct 30 13:02:42.665243 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.665149    4522 scope.go:122] "RemoveContainer" containerID="764a1ce65980d1d46d93c0561626b27d0275e415756884e64775d84280c35fba"
Oct 30 13:02:42.665953 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:02:42.665918    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764a1ce65980d1d46d93c0561626b27d0275e415756884e64775d84280c35fba\": container with ID starting with 764a1ce65980d1d46d93c0561626b27d0275e415756884e64775d84280c35fba not found: ID does not exist" containerID="764a1ce65980d1d46d93c0561626b27d0275e415756884e64775d84280c35fba"
Oct 30 13:02:42.666078 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.665960    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764a1ce65980d1d46d93c0561626b27d0275e415756884e64775d84280c35fba"} err="failed to get container status \"764a1ce65980d1d46d93c0561626b27d0275e415756884e64775d84280c35fba\": rpc error: code = NotFound desc = could not find container \"764a1ce65980d1d46d93c0561626b27d0275e415756884e64775d84280c35fba\": container with ID starting with 764a1ce65980d1d46d93c0561626b27d0275e415756884e64775d84280c35fba not found: ID does not exist"
Oct 30 13:02:42.728104 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728076    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7695fab-0b10-4ffe-857e-32411056876c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7695fab-0b10-4ffe-857e-32411056876c-istiod-ca-cert\") pod \"b7695fab-0b10-4ffe-857e-32411056876c\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") "
Oct 30 13:02:42.728252 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728155    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-istio-token\" (UniqueName: \"kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-istio-token\") pod \"b7695fab-0b10-4ffe-857e-32411056876c\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") "
Oct 30 13:02:42.728252 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728229    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7695fab-0b10-4ffe-857e-32411056876c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7695fab-0b10-4ffe-857e-32411056876c-istio-podinfo\") pod \"b7695fab-0b10-4ffe-857e-32411056876c\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") "
Oct 30 13:02:42.728406 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728285    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-envoy\") pod \"b7695fab-0b10-4ffe-857e-32411056876c\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") "
Oct 30 13:02:42.728406 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728344    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-certs\") pod \"b7695fab-0b10-4ffe-857e-32411056876c\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") "
Oct 30 13:02:42.728538 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728462    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-data\") pod \"b7695fab-0b10-4ffe-857e-32411056876c\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") "
Oct 30 13:02:42.728538 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728494    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-credential-socket\") pod \"b7695fab-0b10-4ffe-857e-32411056876c\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") "
Oct 30 13:02:42.728538 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728528    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-socket\") pod \"b7695fab-0b10-4ffe-857e-32411056876c\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") "
Oct 30 13:02:42.728693 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728577    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-kube-api-access-p8g4g\" (UniqueName: \"kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-kube-api-access-p8g4g\") pod \"b7695fab-0b10-4ffe-857e-32411056876c\" (UID: \"b7695fab-0b10-4ffe-857e-32411056876c\") "
Oct 30 13:02:42.728693 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728571    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7695fab-0b10-4ffe-857e-32411056876c-istiod-ca-cert" pod "b7695fab-0b10-4ffe-857e-32411056876c" (UID: "b7695fab-0b10-4ffe-857e-32411056876c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:42.728805 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728683    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-certs" pod "b7695fab-0b10-4ffe-857e-32411056876c" (UID: "b7695fab-0b10-4ffe-857e-32411056876c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:42.728891 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728872    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-credential-socket" pod "b7695fab-0b10-4ffe-857e-32411056876c" (UID: "b7695fab-0b10-4ffe-857e-32411056876c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:42.728951 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728888    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-data" pod "b7695fab-0b10-4ffe-857e-32411056876c" (UID: "b7695fab-0b10-4ffe-857e-32411056876c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:42.729013 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.728979    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-socket" pod "b7695fab-0b10-4ffe-857e-32411056876c" (UID: "b7695fab-0b10-4ffe-857e-32411056876c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:42.729472 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.729452    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7695fab-0b10-4ffe-857e-32411056876c-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:42.729569 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.729474    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:42.729569 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.729487    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:42.729569 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.729498    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:42.729569 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.729511    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:42.730880 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.730847    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7695fab-0b10-4ffe-857e-32411056876c-istio-podinfo" pod "b7695fab-0b10-4ffe-857e-32411056876c" (UID: "b7695fab-0b10-4ffe-857e-32411056876c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:02:42.730997 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.730960    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-envoy" pod "b7695fab-0b10-4ffe-857e-32411056876c" (UID: "b7695fab-0b10-4ffe-857e-32411056876c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:42.731228 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.731207    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-kube-api-access-p8g4g" pod "b7695fab-0b10-4ffe-857e-32411056876c" (UID: "b7695fab-0b10-4ffe-857e-32411056876c"). InnerVolumeSpecName "kube-api-access-p8g4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:42.731402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.731345    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-istio-token" pod "b7695fab-0b10-4ffe-857e-32411056876c" (UID: "b7695fab-0b10-4ffe-857e-32411056876c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:42.822538 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.822435    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:42.832306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.832277    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7695fab-0b10-4ffe-857e-32411056876c-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:42.832306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.832304    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8g4g\" (UniqueName: \"kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-kube-api-access-p8g4g\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:42.832545 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.832315    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7695fab-0b10-4ffe-857e-32411056876c-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:42.832545 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.832324    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7695fab-0b10-4ffe-857e-32411056876c-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:42.857682 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.857654    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:02:42.920682 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.920653    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:42.933700 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.933660    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-certs\") pod \"a8cf4110-29ff-414d-87af-dfa343a91bc7\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") "
Oct 30 13:02:42.933835 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.933774    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-socket\") pod \"a8cf4110-29ff-414d-87af-dfa343a91bc7\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") "
Oct 30 13:02:42.933896 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.933838    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-data\") pod \"a8cf4110-29ff-414d-87af-dfa343a91bc7\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") "
Oct 30 13:02:42.933896 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.933868    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-podinfo\") pod \"a8cf4110-29ff-414d-87af-dfa343a91bc7\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") "
Oct 30 13:02:42.933998 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.933899    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-token\" (UniqueName: \"kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-token\") pod \"a8cf4110-29ff-414d-87af-dfa343a91bc7\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") "
Oct 30 13:02:42.933998 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.933917    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-certs" pod "a8cf4110-29ff-414d-87af-dfa343a91bc7" (UID: "a8cf4110-29ff-414d-87af-dfa343a91bc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:42.934129 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.934107    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-data" pod "a8cf4110-29ff-414d-87af-dfa343a91bc7" (UID: "a8cf4110-29ff-414d-87af-dfa343a91bc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:42.934180 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.934117    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-envoy\") pod \"a8cf4110-29ff-414d-87af-dfa343a91bc7\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") "
Oct 30 13:02:42.935718 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.934254    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8cf4110-29ff-414d-87af-dfa343a91bc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8cf4110-29ff-414d-87af-dfa343a91bc7-istiod-ca-cert\") pod \"a8cf4110-29ff-414d-87af-dfa343a91bc7\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") "
Oct 30 13:02:42.935718 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.934287    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-credential-socket\") pod \"a8cf4110-29ff-414d-87af-dfa343a91bc7\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") "
Oct 30 13:02:42.935718 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.934365    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-kube-api-access-h5dzh\" (UniqueName: \"kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-kube-api-access-h5dzh\") pod \"a8cf4110-29ff-414d-87af-dfa343a91bc7\" (UID: \"a8cf4110-29ff-414d-87af-dfa343a91bc7\") "
Oct 30 13:02:42.935718 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.934797    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cf4110-29ff-414d-87af-dfa343a91bc7-istiod-ca-cert" pod "a8cf4110-29ff-414d-87af-dfa343a91bc7" (UID: "a8cf4110-29ff-414d-87af-dfa343a91bc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:42.936060 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.935937    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:42.936060 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.935960    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:42.936060 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.935976    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8cf4110-29ff-414d-87af-dfa343a91bc7-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:42.936389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.936239    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-credential-socket" pod "a8cf4110-29ff-414d-87af-dfa343a91bc7" (UID: "a8cf4110-29ff-414d-87af-dfa343a91bc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:42.936894 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.936671    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-socket" pod "a8cf4110-29ff-414d-87af-dfa343a91bc7" (UID: "a8cf4110-29ff-414d-87af-dfa343a91bc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:42.937802 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.937775    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-podinfo" pod "a8cf4110-29ff-414d-87af-dfa343a91bc7" (UID: "a8cf4110-29ff-414d-87af-dfa343a91bc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:02:42.938322 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.938300    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-envoy" pod "a8cf4110-29ff-414d-87af-dfa343a91bc7" (UID: "a8cf4110-29ff-414d-87af-dfa343a91bc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:02:42.938767 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.938734    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-token" pod "a8cf4110-29ff-414d-87af-dfa343a91bc7" (UID: "a8cf4110-29ff-414d-87af-dfa343a91bc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:42.941416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.941383    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-kube-api-access-h5dzh" pod "a8cf4110-29ff-414d-87af-dfa343a91bc7" (UID: "a8cf4110-29ff-414d-87af-dfa343a91bc7"). InnerVolumeSpecName "kube-api-access-h5dzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:42.942790 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.942765    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"]
Oct 30 13:02:42.949487 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:42.949468    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-096cc3-openshift-default-5c98c69cb5-stlql"]
Oct 30 13:02:43.004236 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.004215    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"]
Oct 30 13:02:43.004809 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.004794    4522 cpu_manager.go:414] "containerMap: removing container" podUID="b7695fab-0b10-4ffe-857e-32411056876c" containerName="istio-proxy"
Oct 30 13:02:43.004862 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.004810    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7695fab-0b10-4ffe-857e-32411056876c" containerName="istio-proxy"
Oct 30 13:02:43.004862 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.004825    4522 cpu_manager.go:414] "containerMap: removing container" podUID="b7695fab-0b10-4ffe-857e-32411056876c" containerName="istio-proxy"
Oct 30 13:02:43.004862 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.004833    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7695fab-0b10-4ffe-857e-32411056876c" containerName="istio-proxy"
Oct 30 13:02:43.004862 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.004843    4522 cpu_manager.go:414] "containerMap: removing container" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy"
Oct 30 13:02:43.004862 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.004848    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy"
Oct 30 13:02:43.004862 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.004854    4522 cpu_manager.go:414] "containerMap: removing container" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy"
Oct 30 13:02:43.004862 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.004858    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy"
Oct 30 13:02:43.005077 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.004964    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy"
Oct 30 13:02:43.005077 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.004974    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerName="istio-proxy"
Oct 30 13:02:43.005077 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.004987    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7695fab-0b10-4ffe-857e-32411056876c" containerName="istio-proxy"
Oct 30 13:02:43.005077 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.004995    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7695fab-0b10-4ffe-857e-32411056876c" containerName="istio-proxy"
Oct 30 13:02:43.008566 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.008552    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"
Oct 30 13:02:43.027642 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.027619    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4cb0fe\""
Oct 30 13:02:43.028040 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.027694    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4cb0fe-dockercfg-8627w\""
Oct 30 13:02:43.037068 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.032892    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"]
Oct 30 13:02:43.038069 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.038044    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:43.038180 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.038074    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5dzh\" (UniqueName: \"kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-kube-api-access-h5dzh\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:43.038180 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.038091    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:43.038180 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.038105    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:43.038180 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.038119    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:43.038180 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.038132    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8cf4110-29ff-414d-87af-dfa343a91bc7-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:02:43.086677 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.086651    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"]
Oct 30 13:02:43.091574 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.091556    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"
Oct 30 13:02:43.107711 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.107687    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"]
Oct 30 13:02:43.139884 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.139856    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ckr7\" (UniqueName: \"kubernetes.io/projected/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-kube-api-access-8ckr7\") pod \"payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf\" (UID: \"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"
Oct 30 13:02:43.140019 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.139981    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf\" (UID: \"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"
Oct 30 13:02:43.277673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.274673    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4dcs\" (UniqueName: \"kubernetes.io/projected/b09c05c5-e657-4a02-bdd8-b67bd1449e78-kube-api-access-g4dcs\") pod \"payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx\" (UID: \"b09c05c5-e657-4a02-bdd8-b67bd1449e78\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"
Oct 30 13:02:43.277673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.274922    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ckr7\" (UniqueName: \"kubernetes.io/projected/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-kube-api-access-8ckr7\") pod \"payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf\" (UID: \"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"
Oct 30 13:02:43.277673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.274964    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b09c05c5-e657-4a02-bdd8-b67bd1449e78-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx\" (UID: \"b09c05c5-e657-4a02-bdd8-b67bd1449e78\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"
Oct 30 13:02:43.277673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.275480    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf\" (UID: \"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"
Oct 30 13:02:43.277673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.276475    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf\" (UID: \"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"
Oct 30 13:02:43.385375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.358235    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ckr7\" (UniqueName: \"kubernetes.io/projected/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-kube-api-access-8ckr7\") pod \"payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf\" (UID: \"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"
Oct 30 13:02:43.409373 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.386340    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4dcs\" (UniqueName: \"kubernetes.io/projected/b09c05c5-e657-4a02-bdd8-b67bd1449e78-kube-api-access-g4dcs\") pod \"payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx\" (UID: \"b09c05c5-e657-4a02-bdd8-b67bd1449e78\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"
Oct 30 13:02:43.409373 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.386625    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b09c05c5-e657-4a02-bdd8-b67bd1449e78-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx\" (UID: \"b09c05c5-e657-4a02-bdd8-b67bd1449e78\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"
Oct 30 13:02:43.418838 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.418080    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4dcs\" (UniqueName: \"kubernetes.io/projected/b09c05c5-e657-4a02-bdd8-b67bd1449e78-kube-api-access-g4dcs\") pod \"payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx\" (UID: \"b09c05c5-e657-4a02-bdd8-b67bd1449e78\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"
Oct 30 13:02:43.433857 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.433781    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b09c05c5-e657-4a02-bdd8-b67bd1449e78-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx\" (UID: \"b09c05c5-e657-4a02-bdd8-b67bd1449e78\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"
Oct 30 13:02:43.594013 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.593932    4522 generic.go:356] "Generic (PLEG): container finished" podID="a8cf4110-29ff-414d-87af-dfa343a91bc7" containerID="7dd1d9affb8f4e5a3e94c118c07b17cf2ae2575d91af32d21b99ba66fb3150b8" exitCode=0
Oct 30 13:02:43.594013 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.593987    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"
Oct 30 13:02:43.594231 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.594012    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" event={"ID":"a8cf4110-29ff-414d-87af-dfa343a91bc7","Type":"ContainerDied","Data":"7dd1d9affb8f4e5a3e94c118c07b17cf2ae2575d91af32d21b99ba66fb3150b8"}
Oct 30 13:02:43.594231 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.594064    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m" event={"ID":"a8cf4110-29ff-414d-87af-dfa343a91bc7","Type":"ContainerDied","Data":"787db61112401b81ff9f7dfc40a08cc4faa97b1b7c55218947945437024ad337"}
Oct 30 13:02:43.594231 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.594090    4522 scope.go:122] "RemoveContainer" containerID="7dd1d9affb8f4e5a3e94c118c07b17cf2ae2575d91af32d21b99ba66fb3150b8"
Oct 30 13:02:43.603714 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.603699    4522 scope.go:122] "RemoveContainer" containerID="5aa13ef721e419998aaf634c1f9571846d15e9d37bbd5fdb8598c877335f43ed"
Oct 30 13:02:43.612971 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.612952    4522 scope.go:122] "RemoveContainer" containerID="7dd1d9affb8f4e5a3e94c118c07b17cf2ae2575d91af32d21b99ba66fb3150b8"
Oct 30 13:02:43.613211 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:02:43.613191    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd1d9affb8f4e5a3e94c118c07b17cf2ae2575d91af32d21b99ba66fb3150b8\": container with ID starting with 7dd1d9affb8f4e5a3e94c118c07b17cf2ae2575d91af32d21b99ba66fb3150b8 not found: ID does not exist" containerID="7dd1d9affb8f4e5a3e94c118c07b17cf2ae2575d91af32d21b99ba66fb3150b8"
Oct 30 13:02:43.613297 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.613223    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd1d9affb8f4e5a3e94c118c07b17cf2ae2575d91af32d21b99ba66fb3150b8"} err="failed to get container status \"7dd1d9affb8f4e5a3e94c118c07b17cf2ae2575d91af32d21b99ba66fb3150b8\": rpc error: code = NotFound desc = could not find container \"7dd1d9affb8f4e5a3e94c118c07b17cf2ae2575d91af32d21b99ba66fb3150b8\": container with ID starting with 7dd1d9affb8f4e5a3e94c118c07b17cf2ae2575d91af32d21b99ba66fb3150b8 not found: ID does not exist"
Oct 30 13:02:43.613297 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.613247    4522 scope.go:122] "RemoveContainer" containerID="5aa13ef721e419998aaf634c1f9571846d15e9d37bbd5fdb8598c877335f43ed"
Oct 30 13:02:43.613512 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:02:43.613492    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa13ef721e419998aaf634c1f9571846d15e9d37bbd5fdb8598c877335f43ed\": container with ID starting with 5aa13ef721e419998aaf634c1f9571846d15e9d37bbd5fdb8598c877335f43ed not found: ID does not exist" containerID="5aa13ef721e419998aaf634c1f9571846d15e9d37bbd5fdb8598c877335f43ed"
Oct 30 13:02:43.613554 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.613520    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa13ef721e419998aaf634c1f9571846d15e9d37bbd5fdb8598c877335f43ed"} err="failed to get container status \"5aa13ef721e419998aaf634c1f9571846d15e9d37bbd5fdb8598c877335f43ed\": rpc error: code = NotFound desc = could not find container \"5aa13ef721e419998aaf634c1f9571846d15e9d37bbd5fdb8598c877335f43ed\": container with ID starting with 5aa13ef721e419998aaf634c1f9571846d15e9d37bbd5fdb8598c877335f43ed not found: ID does not exist"
Oct 30 13:02:43.626883 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.626856    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"
Oct 30 13:02:43.650037 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.650004    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"]
Oct 30 13:02:43.655240 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.655213    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e74c27-openshift-default-7b8d554b69-hcw5m"]
Oct 30 13:02:43.705888 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.705637    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"
Oct 30 13:02:43.718754 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.718730    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:43.935801 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:02:43.935772    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059f5d6d_e8a6_4e23_a4d9_ca0153c3638e.slice/crio-e87b8aaa54960157aa268e21b7af0cdd79888a97454029c73860d222c1f91f39 WatchSource:0}: Error finding container e87b8aaa54960157aa268e21b7af0cdd79888a97454029c73860d222c1f91f39: Status 404 returned error can't find the container with id e87b8aaa54960157aa268e21b7af0cdd79888a97454029c73860d222c1f91f39
Oct 30 13:02:43.938574 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:43.938548    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"]
Oct 30 13:02:44.014556 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:02:44.014530    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09c05c5_e657_4a02_bdd8_b67bd1449e78.slice/crio-479b7cb2b7d98c2c485d50b1a0f1d00543cd5c8c3f3eaed3089217aa280d7922 WatchSource:0}: Error finding container 479b7cb2b7d98c2c485d50b1a0f1d00543cd5c8c3f3eaed3089217aa280d7922: Status 404 returned error can't find the container with id 479b7cb2b7d98c2c485d50b1a0f1d00543cd5c8c3f3eaed3089217aa280d7922
Oct 30 13:02:44.017996 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:44.017866    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"]
Oct 30 13:02:44.336902 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:44.336870    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8cf4110-29ff-414d-87af-dfa343a91bc7" path="/var/lib/kubelet/pods/a8cf4110-29ff-414d-87af-dfa343a91bc7/volumes"
Oct 30 13:02:44.337341 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:44.337326    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7695fab-0b10-4ffe-857e-32411056876c" path="/var/lib/kubelet/pods/b7695fab-0b10-4ffe-857e-32411056876c/volumes"
Oct 30 13:02:44.601998 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:44.601910    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx" event={"ID":"b09c05c5-e657-4a02-bdd8-b67bd1449e78","Type":"ContainerStarted","Data":"37e89a5725b295e0c922b0f7b679300e028d95b2998e9428d04cd1d08c704b3c"}
Oct 30 13:02:44.601998 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:44.601968    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx" event={"ID":"b09c05c5-e657-4a02-bdd8-b67bd1449e78","Type":"ContainerStarted","Data":"479b7cb2b7d98c2c485d50b1a0f1d00543cd5c8c3f3eaed3089217aa280d7922"}
Oct 30 13:02:44.601998 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:44.601994    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"
Oct 30 13:02:44.604497 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:44.604473    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf" event={"ID":"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e","Type":"ContainerStarted","Data":"b125331ee9343588270b5dfb619b4e8d70f69091bb2ba2636c904dfc37e5259e"}
Oct 30 13:02:44.604584 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:44.604512    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf" event={"ID":"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e","Type":"ContainerStarted","Data":"e87b8aaa54960157aa268e21b7af0cdd79888a97454029c73860d222c1f91f39"}
Oct 30 13:02:44.604584 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:44.604548    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"
Oct 30 13:02:44.641278 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:44.641236    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx" podStartSLOduration=1.64121961 podStartE2EDuration="1.64121961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:44.640616109 +0000 UTC m=+1608.455942600" watchObservedRunningTime="2026-10-30 13:02:44.64121961 +0000 UTC m=+1608.456546099"
Oct 30 13:02:44.679676 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:44.679637    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf" podStartSLOduration=2.679627069 podStartE2EDuration="2.679627069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:44.677574899 +0000 UTC m=+1608.492901385" watchObservedRunningTime="2026-10-30 13:02:44.679627069 +0000 UTC m=+1608.494953600"
Oct 30 13:02:45.557807 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:45.557776    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:47.219335 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:47.219300    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:48.900525 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:48.900496    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:49.066401 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:49.065696    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:50.560064 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:50.560032    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:51.134185 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:02:51.134149    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:02:52.901106 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:52.901054    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:54.627281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:54.627234    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:55.635128 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:55.635101    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"
Oct 30 13:02:55.635546 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:55.635164    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"
Oct 30 13:02:56.730995 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:56.730957    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:56.815501 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:56.815471    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:56.921457 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:56.921410    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:58.537247 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:02:58.537218    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:00.846822 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:00.846787    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"]
Oct 30 13:03:00.852333 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:00.852311    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:00.868124 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:00.868101    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-4cb0fe-serving-cert\""
Oct 30 13:03:00.878987 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:00.878956    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"]
Oct 30 13:03:00.939214 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:00.939179    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-487z7\" (UniqueName: \"kubernetes.io/projected/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-kube-api-access-487z7\") pod \"maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:00.939434 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:00.939262    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:00.939434 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:00.939339    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:00.939434 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:00.939406    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-metrics-certs\") pod \"maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:00.979763 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:00.979729    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:01.040476 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:01.040447    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487z7\" (UniqueName: \"kubernetes.io/projected/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-kube-api-access-487z7\") pod \"maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:01.040658 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:01.040521    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:01.040658 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:01.040594    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:01.040658 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:01.040615    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-metrics-certs\") pod \"maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:01.041324 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:01.041298    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:01.043063 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:01.043044    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:01.043169 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:01.043150    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-metrics-certs\") pod \"maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:01.128298 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:01.128199    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-487z7\" (UniqueName: \"kubernetes.io/projected/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-kube-api-access-487z7\") pod \"maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:01.178530 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:01.178496    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:01.181275 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:01.181135    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:01.532176 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:03:01.532139    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186f7bca_cdb1_44a8_b70d_3f02a9c891a8.slice/crio-6429f217acfd40cb1e3ef856f3799023f9e5ac5b048f9686e66bb42bf23d4cc4 WatchSource:0}: Error finding container 6429f217acfd40cb1e3ef856f3799023f9e5ac5b048f9686e66bb42bf23d4cc4: Status 404 returned error can't find the container with id 6429f217acfd40cb1e3ef856f3799023f9e5ac5b048f9686e66bb42bf23d4cc4
Oct 30 13:03:01.533743 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:01.533716    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"]
Oct 30 13:03:01.533910 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:01.533896    4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:03:01.844864 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:01.844781    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr" event={"ID":"186f7bca-cdb1-44a8-b70d-3f02a9c891a8","Type":"ContainerStarted","Data":"6429f217acfd40cb1e3ef856f3799023f9e5ac5b048f9686e66bb42bf23d4cc4"}
Oct 30 13:03:02.861211 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:02.861169    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr" event={"ID":"186f7bca-cdb1-44a8-b70d-3f02a9c891a8","Type":"ContainerStarted","Data":"c6f96187819a70ba40fdd928bd06de9edf7f2a14585cc95e855f90379987db07"}
Oct 30 13:03:02.861663 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:02.861266    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:02.921859 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:02.921809    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr" podStartSLOduration=2.520499697 podStartE2EDuration="2.921796341s" totalImagesPullingTime="401.296644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:02.919532587 +0000 UTC m=+1626.734859076" watchObservedRunningTime="2026-10-30 13:03:02.921796341 +0000 UTC m=+1626.737122830"
Oct 30 13:03:03.232211 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:03.232119    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:03.404635 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:03.404580    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:05.283264 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:05.283176    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:07.020536 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:07.020493    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:08.872613 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:08.872570    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:03:09.335941 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:09.335890    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:11.142161 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:11.142127    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:12.896768 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:12.896729    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:14.807540 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:14.807504    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:16.673660 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:16.672827    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 13:03:17.701228 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:17.701190    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:18.372491 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:18.372458    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"]
Oct 30 13:03:18.542858 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:18.542784    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:18.739936 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:18.739849    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:20.703377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:20.700163    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:21.298952 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:03:21.298908    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:03:21.987380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:21.986490    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:22.554389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:22.554116    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:22.894485 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:22.889302    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:03:23.186436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:23.185496    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:23.376471 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:23.376421    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w" podUID="df3a485d-9d13-4d54-b025-4ee740558a4b" containerName="payload-pre-processing" containerID="cri-o://7b7ff3fe9800bffd11c468989d1373fdd15a0c90063e9e3a23b793a0828bf3af" gracePeriod=25
Oct 30 13:03:23.493816 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:23.488968    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:23.581387 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:23.546005    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:23.587380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:23.569287    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:23.587380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:23.587296    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:03:23.699379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:23.686446    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:23.871382 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:23.866141    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:23.927379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:23.902436    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:23.928141 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:23.928113    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:03:23.971795 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:23.929579    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:23.971795 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:23.953427    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.058380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.057977    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.077532 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.077459    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.220436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.189291    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.240604 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.240563    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:03:24.297430 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.249191    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 30 13:03:24.497776 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.491431    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.573409 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.561419    4522 generic.go:356] "Generic (PLEG): container finished" podID="df3a485d-9d13-4d54-b025-4ee740558a4b" containerID="7b7ff3fe9800bffd11c468989d1373fdd15a0c90063e9e3a23b793a0828bf3af" exitCode=0
Oct 30 13:03:24.704379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.703700    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.731230 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.731186    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:03:24.737385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.732269    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.778982 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.778912    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:03:24.779275 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.778958    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w" event={"ID":"df3a485d-9d13-4d54-b025-4ee740558a4b","Type":"ContainerDied","Data":"7b7ff3fe9800bffd11c468989d1373fdd15a0c90063e9e3a23b793a0828bf3af"}
Oct 30 13:03:24.807729 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.807653    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:03:24.808319 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.808277    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:03:24.809546 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.809507    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded"
Oct 30 13:03:24.866001 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.865967    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:24.866286 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.866251    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.899537 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.873642    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.937406 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.933627    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.937406 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.935864    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:03:24.937843 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.937806    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:03:24.962588 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.962529    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.976022 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.963525    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:03:24.976022 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.975918    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.984575 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.984524    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:24.985345 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:24.985323    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:03:25.147781 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:25.147675    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:03:25.218626 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:25.218587    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 13:03:25.248510 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:25.248463    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:25.250602 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:25.250573    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:03:25.462788 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:25.462665    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:03:25.586066 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:25.560475    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:25.806438 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:25.714696    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:25.806438 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:25.777630    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded"
Oct 30 13:03:25.848052 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:25.848017    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:25.891528 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:25.887487    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:26.032983 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.032931    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:26.035517 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.035460    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:26.061101 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.060696    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:26.062121 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:25.848320    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:26.068570 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.068516    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:03:26.115596 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.115551    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:26.146949 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.146890    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:26.251459 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.148459    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:03:26.334717 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.265598    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:26.400378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.395369    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:03:26.463816 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.463760    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:03:26.715367 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.715250    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:03:26.746724 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.746685    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:26.905380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.877383    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:26.918811 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.918776    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"
Oct 30 13:03:26.985565 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:26.985463    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:03:27.065388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.051006    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df3a485d-9d13-4d54-b025-4ee740558a4b-kube-api-access-bqvhp\" (UniqueName: \"kubernetes.io/projected/df3a485d-9d13-4d54-b025-4ee740558a4b-kube-api-access-bqvhp\") pod \"df3a485d-9d13-4d54-b025-4ee740558a4b\" (UID: \"df3a485d-9d13-4d54-b025-4ee740558a4b\") "
Oct 30 13:03:27.094593 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.094558    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df3a485d-9d13-4d54-b025-4ee740558a4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3a485d-9d13-4d54-b025-4ee740558a4b-plugins-config-volume\") pod \"df3a485d-9d13-4d54-b025-4ee740558a4b\" (UID: \"df3a485d-9d13-4d54-b025-4ee740558a4b\") "
Oct 30 13:03:27.107386 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.076370    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:27.107952 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.076296    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:27.108499 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.108476    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:03:27.114480 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.095184    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3a485d-9d13-4d54-b025-4ee740558a4b-plugins-config-volume" pod "df3a485d-9d13-4d54-b025-4ee740558a4b" (UID: "df3a485d-9d13-4d54-b025-4ee740558a4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:27.134388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.132470    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:27.134388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.132533    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:27.134388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.133160    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:27.233601 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.233557    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3a485d-9d13-4d54-b025-4ee740558a4b-kube-api-access-bqvhp" pod "df3a485d-9d13-4d54-b025-4ee740558a4b" (UID: "df3a485d-9d13-4d54-b025-4ee740558a4b"). InnerVolumeSpecName "kube-api-access-bqvhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:27.249437 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.235698    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df3a485d-9d13-4d54-b025-4ee740558a4b-kube-api-access-bqvhp\" (UniqueName: \"kubernetes.io/projected/df3a485d-9d13-4d54-b025-4ee740558a4b-kube-api-access-bqvhp\") pod \"df3a485d-9d13-4d54-b025-4ee740558a4b\" (UID: \"df3a485d-9d13-4d54-b025-4ee740558a4b\") "
Oct 30 13:03:27.249437 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.236519    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3a485d-9d13-4d54-b025-4ee740558a4b-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:03:27.249437 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:03:27.236616    4522 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/df3a485d-9d13-4d54-b025-4ee740558a4b/volumes/kubernetes.io~projected/kube-api-access-bqvhp
Oct 30 13:03:27.249437 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.236664    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3a485d-9d13-4d54-b025-4ee740558a4b-kube-api-access-bqvhp" pod "df3a485d-9d13-4d54-b025-4ee740558a4b" (UID: "df3a485d-9d13-4d54-b025-4ee740558a4b"). InnerVolumeSpecName "kube-api-access-bqvhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:27.258511 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.258471    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:27.283414 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.261434    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:27.422418 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.347307    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:27.440779 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.348306    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqvhp\" (UniqueName: \"kubernetes.io/projected/df3a485d-9d13-4d54-b025-4ee740558a4b-kube-api-access-bqvhp\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:03:27.441015 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.431204    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:03:27.526167 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.526124    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 13:03:27.564918 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.564869    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:03:27.678374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.674339    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:27.678374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.674428    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:27.722389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.719707    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:03:27.738438 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.737884    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded"
Oct 30 13:03:27.747219 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:27.747166    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:28.068379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.063465    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:28.124497 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.124448    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:03:28.125863 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.125316    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:03:28.157473 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.157425    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:03:28.190291 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.190259    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"
Oct 30 13:03:28.191003 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.190973    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w" event={"ID":"df3a485d-9d13-4d54-b025-4ee740558a4b","Type":"ContainerDied","Data":"bab57b5ab758d7c0556d6209c0a5d6302fea2522c0d2399122966d727441ef3e"}
Oct 30 13:03:28.191194 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.191178    4522 scope.go:122] "RemoveContainer" containerID="7b7ff3fe9800bffd11c468989d1373fdd15a0c90063e9e3a23b793a0828bf3af"
Oct 30 13:03:28.285379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.281655    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:03:28.331482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.286811    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:03:28.443503 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.443455    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:28.508380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.487274    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:28.587451 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.565259    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:28.722306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.722264    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:28.766378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.764270    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:28.814516 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.814418    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"]
Oct 30 13:03:28.830431 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:28.822929    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-87f1b5-7947d986c-vqk5w"]
Oct 30 13:03:29.112493 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.112445    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:29.125526 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.125464    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:03:29.126119 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.126084    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:03:29.159493 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.159444    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:03:29.328495 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.328445    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:29.346499 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.335539    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:03:29.348379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.348326    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:29.444429 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.444097    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:29.489270 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.489228    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:03:29.520731 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.520687    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:29.566476 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.566432    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:29.759318 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.759184    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:03:29.765456 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.765417    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:29.853675 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.853641    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:03:29.912513 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.910762    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:03:29.946091 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:29.946063    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:03:30.047846 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:30.047818    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:03:30.065483 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:30.065463    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 13:03:30.085492 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:30.085468    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:03:30.126785 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:30.126737    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:03:30.130617 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:30.130594    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:03:30.133043 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:30.133012    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:30.136554 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:30.136495    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:03:30.155644 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:30.155624    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:03:30.156701 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:30.156676    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:03:30.162188 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:30.162167    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:03:30.339847 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:30.339753    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df3a485d-9d13-4d54-b025-4ee740558a4b" path="/var/lib/kubelet/pods/df3a485d-9d13-4d54-b025-4ee740558a4b/volumes"
Oct 30 13:03:31.566398 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:31.566203    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:32.602496 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:32.602459    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:33.611374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:33.603847    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:33.824688 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:33.824657    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:34.466380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:34.465952    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:34.659587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:34.659552    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:35.463376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:35.463318    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:36.917529 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:36.917494    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:37.438804 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:37.438765    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:38.060198 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.060166    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:38.175385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.175343    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:38.376403 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.376334    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:38.415775 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.415749    4522 generic.go:356] "Generic (PLEG): container finished" podID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerID="a88c70190f694f652b69b8a9cd45b6c6843b3520f6166ac501897ce398e7c28c" exitCode=0
Oct 30 13:03:38.415931 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.415792    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" event={"ID":"649cac72-df6c-4572-8fc9-a3e78a5e20b8","Type":"ContainerDied","Data":"a88c70190f694f652b69b8a9cd45b6c6843b3520f6166ac501897ce398e7c28c"}
Oct 30 13:03:38.416133 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.416116    4522 scope.go:122] "RemoveContainer" containerID="a88c70190f694f652b69b8a9cd45b6c6843b3520f6166ac501897ce398e7c28c"
Oct 30 13:03:38.417619 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.417587    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:38.417720 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.417657    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:38.417720 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.417691    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:38.591273 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.591235    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"]
Oct 30 13:03:38.592409 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.592384    4522 cpu_manager.go:414] "containerMap: removing container" podUID="df3a485d-9d13-4d54-b025-4ee740558a4b" containerName="payload-pre-processing"
Oct 30 13:03:38.592595 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.592580    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df3a485d-9d13-4d54-b025-4ee740558a4b" containerName="payload-pre-processing"
Oct 30 13:03:38.592949 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.592923    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="df3a485d-9d13-4d54-b025-4ee740558a4b" containerName="payload-pre-processing"
Oct 30 13:03:38.600065 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.597081    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.621444 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.620375    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-8f3cb8-openshift-default-dockercfg-5ntvl\""
Oct 30 13:03:38.628011 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.626273    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"]
Oct 30 13:03:38.669064 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.669036    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:38.728159 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.728129    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgbn\" (UniqueName: \"kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-kube-api-access-dtgbn\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.728377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.728164    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-socket\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.728377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.728293    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-envoy\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.728377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.728337    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-certs\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.728573 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.728448    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-token\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.728573 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.728510    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-credential-socket\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.728573 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.728558    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-podinfo\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.728745 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.728583    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istiod-ca-cert\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.728745 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.728602    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-data\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.833398 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.828906    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:38.853375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.848143    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-credential-socket\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.853375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.848221    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-podinfo\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.858684 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.858641    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-credential-socket\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.858862 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.858757    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istiod-ca-cert\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.858862 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.858798    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-data\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.858993 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.858884    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgbn\" (UniqueName: \"kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-kube-api-access-dtgbn\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.858993 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.858922    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-socket\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.859109 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.859061    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-envoy\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.859172 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.859114    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-certs\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.859284 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.859264    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-token\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.867388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.860383    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istiod-ca-cert\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.867388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.860642    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-data\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.867388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.861099    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-socket\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.875375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.868941    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-podinfo\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.875851 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.875829    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-certs\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.908418 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.906073    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgbn\" (UniqueName: \"kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-kube-api-access-dtgbn\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.908418 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.906621    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-token\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.909066 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.908749    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-envoy\") pod \"e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") " pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:38.936375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:38.934658    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:39.424074 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:39.424044    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" event={"ID":"649cac72-df6c-4572-8fc9-a3e78a5e20b8","Type":"ContainerStarted","Data":"547c2f1fe06583e214b5623cc493e1d530d569cb5b78018a8fb1f6b33e300898"}
Oct 30 13:03:39.465046 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:39.465013    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"]
Oct 30 13:03:39.472389 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:03:39.467329    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb69b573_fa0e_4789_80e7_8df8caef1cb7.slice/crio-5943893e5891fbe1a3e72c1a6825b9a0115b0d28e9d281c7c684652600e56e60 WatchSource:0}: Error finding container 5943893e5891fbe1a3e72c1a6825b9a0115b0d28e9d281c7c684652600e56e60: Status 404 returned error can't find the container with id 5943893e5891fbe1a3e72c1a6825b9a0115b0d28e9d281c7c684652600e56e60
Oct 30 13:03:39.472389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:39.470620    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:39.472389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:39.470693    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:39.472389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:39.470735    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:39.563889 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:39.563860    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:39.640750 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:39.640718    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:03:39.661311 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:39.661240    4522 patch_prober.go:29] interesting pod/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 13:03:39.661670 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:39.661609    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 13:03:40.107923 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:40.107891    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:40.238551 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:40.238514    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:40.510424 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:40.507799    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:03:40.641891 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:40.641851    4522 patch_prober.go:29] interesting pod/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 13:03:40.654381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:40.646526    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 13:03:40.703379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:40.694138    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" event={"ID":"cb69b573-fa0e-4789-80e7-8df8caef1cb7","Type":"ContainerStarted","Data":"78065e469c98a58f34a7f18f8c4890205c0af25924b3ae184c8b93a90631063a"}
Oct 30 13:03:40.703379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:40.694199    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" event={"ID":"cb69b573-fa0e-4789-80e7-8df8caef1cb7","Type":"ContainerStarted","Data":"5943893e5891fbe1a3e72c1a6825b9a0115b0d28e9d281c7c684652600e56e60"}
Oct 30 13:03:40.835687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:40.834586    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" podStartSLOduration=2.8345666940000003 podStartE2EDuration="2.834566694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:40.832706468 +0000 UTC m=+1664.648032960" watchObservedRunningTime="2026-10-30 13:03:40.834566694 +0000 UTC m=+1664.649893186"
Oct 30 13:03:40.961779 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:40.961738    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:40.996379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:40.992754    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:41.328264 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:41.328229    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:41.651215 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:41.644244    4522 patch_prober.go:29] interesting pod/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 13:03:41.651215 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:41.644316    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 13:03:41.779263 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:41.773553    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:41.779263 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:41.773734    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:03:42.181490 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:42.178401    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:42.413380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:42.407221    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:42.558375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:42.554032    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:42.640570 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:42.640537    4522 patch_prober.go:29] interesting pod/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 13:03:42.640732 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:42.640585    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 13:03:42.822912 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:42.822806    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:43.641195 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:43.641163    4522 patch_prober.go:29] interesting pod/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 13:03:43.641398 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:43.641223    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 13:03:43.700430 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:43.698745    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:43.894307 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:43.894223    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:44.549547 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:44.549513    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:44.650406 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:44.642046    4522 patch_prober.go:29] interesting pod/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 13:03:44.650406 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:44.642109    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 13:03:45.393239 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:45.393207    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:45.506308 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:45.506276    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:45.594525 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:45.594493    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"]
Oct 30 13:03:45.599589 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:45.599567    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"
Oct 30 13:03:45.625971 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:45.625945    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-8f3cb8-dockercfg-pvqgx\""
Oct 30 13:03:45.626110 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:45.625995    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-8f3cb8\""
Oct 30 13:03:45.632889 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:45.631983    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"]
Oct 30 13:03:45.649739 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:45.648678    4522 patch_prober.go:29] interesting pod/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 13:03:45.650059 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:45.650010    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 13:03:45.723382 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:45.707500    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2ts2\" (UniqueName: \"kubernetes.io/projected/9a297ff9-2434-4474-8403-a57b1fb5f8ec-kube-api-access-w2ts2\") pod \"payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c\" (UID: \"9a297ff9-2434-4474-8403-a57b1fb5f8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"
Oct 30 13:03:45.747382 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:45.723725    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a297ff9-2434-4474-8403-a57b1fb5f8ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c\" (UID: \"9a297ff9-2434-4474-8403-a57b1fb5f8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"
Oct 30 13:03:45.924439 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:45.914171    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"]
Oct 30 13:03:45.990173 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:45.989757    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"
Oct 30 13:03:46.000090 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.000052    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"]
Oct 30 13:03:46.020137 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.020098    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ts2\" (UniqueName: \"kubernetes.io/projected/9a297ff9-2434-4474-8403-a57b1fb5f8ec-kube-api-access-w2ts2\") pod \"payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c\" (UID: \"9a297ff9-2434-4474-8403-a57b1fb5f8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"
Oct 30 13:03:46.020291 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.020178    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xdp\" (UniqueName: \"kubernetes.io/projected/d2e6e069-7ef4-4e12-b266-6f1316d1878d-kube-api-access-w2xdp\") pod \"payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j\" (UID: \"d2e6e069-7ef4-4e12-b266-6f1316d1878d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"
Oct 30 13:03:46.020291 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.020252    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a297ff9-2434-4474-8403-a57b1fb5f8ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c\" (UID: \"9a297ff9-2434-4474-8403-a57b1fb5f8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"
Oct 30 13:03:46.020629 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.020468    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e6e069-7ef4-4e12-b266-6f1316d1878d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j\" (UID: \"d2e6e069-7ef4-4e12-b266-6f1316d1878d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"
Oct 30 13:03:46.021571 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.021540    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a297ff9-2434-4474-8403-a57b1fb5f8ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c\" (UID: \"9a297ff9-2434-4474-8403-a57b1fb5f8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"
Oct 30 13:03:46.048004 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.047976    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ts2\" (UniqueName: \"kubernetes.io/projected/9a297ff9-2434-4474-8403-a57b1fb5f8ec-kube-api-access-w2ts2\") pod \"payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c\" (UID: \"9a297ff9-2434-4474-8403-a57b1fb5f8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"
Oct 30 13:03:46.122533 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.122497    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e6e069-7ef4-4e12-b266-6f1316d1878d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j\" (UID: \"d2e6e069-7ef4-4e12-b266-6f1316d1878d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"
Oct 30 13:03:46.122747 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.122676    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xdp\" (UniqueName: \"kubernetes.io/projected/d2e6e069-7ef4-4e12-b266-6f1316d1878d-kube-api-access-w2xdp\") pod \"payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j\" (UID: \"d2e6e069-7ef4-4e12-b266-6f1316d1878d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"
Oct 30 13:03:46.123400 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.123374    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e6e069-7ef4-4e12-b266-6f1316d1878d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j\" (UID: \"d2e6e069-7ef4-4e12-b266-6f1316d1878d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"
Oct 30 13:03:46.150495 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.150470    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xdp\" (UniqueName: \"kubernetes.io/projected/d2e6e069-7ef4-4e12-b266-6f1316d1878d-kube-api-access-w2xdp\") pod \"payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j\" (UID: \"d2e6e069-7ef4-4e12-b266-6f1316d1878d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"
Oct 30 13:03:46.213758 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.213686    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"
Oct 30 13:03:46.329403 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.328627    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"
Oct 30 13:03:46.529738 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:03:46.529708    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a297ff9_2434_4474_8403_a57b1fb5f8ec.slice/crio-30c53226c79b173b5d2e418625186fa8b68622824afa64e510454ccac938a063 WatchSource:0}: Error finding container 30c53226c79b173b5d2e418625186fa8b68622824afa64e510454ccac938a063: Status 404 returned error can't find the container with id 30c53226c79b173b5d2e418625186fa8b68622824afa64e510454ccac938a063
Oct 30 13:03:46.530720 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.530695    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"]
Oct 30 13:03:46.641009 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:03:46.640977    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e6e069_7ef4_4e12_b266_6f1316d1878d.slice/crio-9e95e1beadea3f355884dcdb0558e8161b4785ef4c8e340af31cab372fec068c WatchSource:0}: Error finding container 9e95e1beadea3f355884dcdb0558e8161b4785ef4c8e340af31cab372fec068c: Status 404 returned error can't find the container with id 9e95e1beadea3f355884dcdb0558e8161b4785ef4c8e340af31cab372fec068c
Oct 30 13:03:46.642377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.642339    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"]
Oct 30 13:03:46.646244 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.646223    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:03:46.647303 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.647285    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:03:46.917710 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.917614    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c" event={"ID":"9a297ff9-2434-4474-8403-a57b1fb5f8ec","Type":"ContainerStarted","Data":"62680a7039803ea170879f6c8715d639c2c666fe245eccfa523c7a8e326ca3e0"}
Oct 30 13:03:46.917710 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.917674    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c" event={"ID":"9a297ff9-2434-4474-8403-a57b1fb5f8ec","Type":"ContainerStarted","Data":"30c53226c79b173b5d2e418625186fa8b68622824afa64e510454ccac938a063"}
Oct 30 13:03:46.917710 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.917699    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"
Oct 30 13:03:46.919224 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.919199    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j" event={"ID":"d2e6e069-7ef4-4e12-b266-6f1316d1878d","Type":"ContainerStarted","Data":"d86f058d2a5d33e08057396b51f8b71efe33e23fda505cf2b17c47df31dcde78"}
Oct 30 13:03:46.919348 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.919240    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j" event={"ID":"d2e6e069-7ef4-4e12-b266-6f1316d1878d","Type":"ContainerStarted","Data":"9e95e1beadea3f355884dcdb0558e8161b4785ef4c8e340af31cab372fec068c"}
Oct 30 13:03:46.958251 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.958201    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c" podStartSLOduration=1.9581847909999999 podStartE2EDuration="1.958184791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:46.956396876 +0000 UTC m=+1670.771723373" watchObservedRunningTime="2026-10-30 13:03:46.958184791 +0000 UTC m=+1670.773511281"
Oct 30 13:03:46.999634 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:46.999596    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j" podStartSLOduration=1.9995845079999999 podStartE2EDuration="1.999584508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:46.996378949 +0000 UTC m=+1670.811705457" watchObservedRunningTime="2026-10-30 13:03:46.999584508 +0000 UTC m=+1670.814911015"
Oct 30 13:03:47.133247 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:47.133223    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:47.925759 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:47.925728    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"
Oct 30 13:03:48.973411 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:48.973381    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:49.151371 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:49.150391    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:49.271519 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:49.268176    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:49.455021 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:49.454982    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:49.601433 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:49.590347    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:50.112738 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:50.105335    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:50.487445 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:50.484532    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:51.136380 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:03:51.136314    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:03:51.523437 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:51.522837    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:53.423366 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:53.423268    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:53.651990 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:53.651949    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:53.878582 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:53.878551    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:55.336900 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:55.336864    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:55.466375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:55.458778    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:55.563966 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:55.563933    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:57.419647 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:57.419612    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:57.928502 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:57.928472    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"
Oct 30 13:03:58.934673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:58.934644    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"
Oct 30 13:03:59.138629 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:03:59.138597    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:00.915957 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:00.915920    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:01.020318 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:01.020289    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:01.129929 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:01.129890    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:02.806045 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:02.805998    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:04.589459 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:04.589425    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:05.298079 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:05.298042    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:05.422906 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:05.422851    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:06.367096 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:06.367039    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:06.457215 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:06.457182    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:06.600411 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:06.600347    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:08.226373 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:08.226314    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:10.163333 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:10.163299    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:12.631448 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:12.631410    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:14.451385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:14.449541    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:16.709196 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:16.709141    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:18.461050 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:18.461017    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:18.560030 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:18.560002    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:18.835323 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:18.835292    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:20.158235 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:20.158188    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:20.728903 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:20.728864    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:20.877380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:20.877279    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:20.985627 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:20.985508    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:21.135424 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:04:21.135387    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:04:21.458977 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:21.458942    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:23.050675 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:23.050645    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:25.145954 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:25.145905    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:27.026423 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:27.023588    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:28.326990 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:28.326950    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:28.464483 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:28.463106    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:28.836441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:28.821877    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:28.935378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:28.933989    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:30.985010 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:30.984970    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:32.720418 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:32.720386    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:33.172553 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:33.172518    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 13:04:34.200129 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:34.200085    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:34.667289 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:34.667230    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:37.089375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:37.086196    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:37.126387 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:37.123458    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:37.130376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:37.128429    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:04:37.130376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:37.129407    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:37.281835 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:37.281800    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:37.293593 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:37.293515    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:04:37.324949 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:37.324818    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:37.964997 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:37.962950    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:37.964997 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:37.963852    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:04:37.964997 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:37.964622    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:37.964997 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:37.964934    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 13:04:37.970312 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:37.970278    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:37.970625 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:37.970554    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:38.123775 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:38.123732    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:38.124980 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:38.124956    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:04:38.149816 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:38.149781    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:38.150654 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:38.150629    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:04:38.298388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:38.297461    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:38.300382 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:38.299843    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:04:38.435389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:38.431726    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:38.435389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:38.431996    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:38.435389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:38.432491    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 13:04:38.958497 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:38.920476    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:04:38.971379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:38.963719    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:38.971379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:38.964171    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:39.010378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.010288    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:39.033705 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.011757    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:39.143584 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.143532    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:39.158417 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.149418    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:04:39.175756 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.150217    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:39.351673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.340303    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:04:39.351673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.350232    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:04:39.351673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.350322    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:04:39.351673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.350394    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:04:39.351673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.350432    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:04:39.389850 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.389813    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:39.540511 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.540458    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:39.541724 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.541697    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:04:39.562718 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.562677    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:39.658521 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.658413    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded"
Oct 30 13:04:39.872224 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.872188    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:39.900965 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.900933    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:39.901244 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.901210    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:39.934424 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.930910    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:04:39.955882 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:39.955848    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:04:40.000795 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.000745    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:40.001368 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.001320    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:04:40.066210 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.054921    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:04:40.072658 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.072604    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:04:40.158404 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.157149    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:40.163382 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.158935    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:04:40.192432 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.188495    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:40.192432 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.189789    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:40.358494 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.358445    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:04:40.549691 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.549645    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:40.655202 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.624142    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:40.825815 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.825725    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:04:40.953010 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.952964    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:40.953674 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:40.953638    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:41.001738 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.001688    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:41.092974 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.092869    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:41.097406 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.093599    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:41.439773 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.225593    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:04:41.442692 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.254133    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:04:41.451122 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.256707    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:04:41.479599 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.286370    4522 patch_prober.go:29] interesting pod/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:41.480736 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.480701    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:41.482186 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.309776    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:41.487883 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.482455    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:41.488367 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.486815    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:04:41.499935 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.486857    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:04:41.539379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.535773    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:41.632390 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.632326    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:04:41.650385 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:04:41.649460    4522 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.287s"
Oct 30 13:04:41.850756 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.850705    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:41.973377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.972488    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:41.973377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:41.972575    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:42.015389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:42.011581    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:42.098384 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:42.097479    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:42.431282 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:42.431211    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:42.654402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:42.446554    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:42.654402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:42.446914    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:04:42.654402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:42.473927    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:42.654402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:42.502109    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:42.654402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:42.503080    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:42.654402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:42.503376    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:04:42.654402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:42.587520    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:42.654402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:42.652440    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:04:42.935511 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:42.935465    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:04:43.032045 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:43.031444    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:43.106656 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:43.105572    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:43.437254 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:43.436443    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:04:43.498092 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:43.493835    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:43.498092 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:43.494188    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:04:43.498092 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:43.497137    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:43.506615 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:43.505427    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:43.506615 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:43.505752    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:04:43.506615 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:43.506022    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:04:43.655380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:43.653546    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:04:43.937324 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:43.936878    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:04:44.051378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.035369    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:44.183378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.143951    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:44.190619 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.190527    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:04:44.246819 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.246769    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:44.460308 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.459844    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:44.549728 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.549678    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:44.551215 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.551174    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:04:44.552487 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.552463    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:04:44.578522 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.578474    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:44.632973 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.586213    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:44.634417 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.587062    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:44.712712 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.712625    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:04:44.721843 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.721821    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:04:44.735126 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.735106    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:04:44.815038 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.814915    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded"
Oct 30 13:04:44.865000 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.864966    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:04:44.940809 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.940767    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 13:04:44.969990 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.967204    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:04:44.983414 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:44.979737    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Oct 30 13:04:45.101379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:45.073977    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:45.154582 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:45.154540    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:04:45.288568 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:45.288542    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:04:45.294900 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:45.294859    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:45.314181 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:45.304432    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:04:45.322929 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:45.322890    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 13:04:45.324917 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:45.324888    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:04:45.326711 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:45.326682    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:04:46.924330 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:46.924292    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:49.385430 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:49.384475    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:49.480761 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:49.480727    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:49.679735 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:49.679634    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hv6zx_4098fe5a-e7d3-488c-a1ed-72d447999e15/payload-processing/0.log"
Oct 30 13:04:49.904325 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:49.904296    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hv6zx_4098fe5a-e7d3-488c-a1ed-72d447999e15/payload-processing/0.log"
Oct 30 13:04:50.390905 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.390866    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"]
Oct 30 13:04:50.395238 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.395214    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.409861 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.409838    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-f73c9f-openshift-default-dockercfg-zj7p2\""
Oct 30 13:04:50.425730 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.425703    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"]
Oct 30 13:04:50.531564 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.531527    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-podinfo\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.531705 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.531593    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-token\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.531751 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.531698    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-socket\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.531751 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.531734    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istiod-ca-cert\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.531841 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.531752    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-data\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.531841 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.531805    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-credential-socket\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.531942 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.531925    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-certs\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.532001 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.531989    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csmg6\" (UniqueName: \"kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-kube-api-access-csmg6\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.532042 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.532031    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-envoy\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.633749 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.633724    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-certs\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.633893 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.633790    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csmg6\" (UniqueName: \"kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-kube-api-access-csmg6\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.633893 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.633858    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-envoy\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.633989 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.633915    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-podinfo\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.633989 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.633971    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-token\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.634102 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.634042    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-socket\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.634102 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.634088    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istiod-ca-cert\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.634211 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.634114    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-data\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.634211 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.634177    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-certs\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.634324 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.634139    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-credential-socket\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.634759 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.634466    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-socket\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.634759 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.634535    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-data\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.634759 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.634664    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-credential-socket\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.634967 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.634846    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istiod-ca-cert\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.636640 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.636618    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-envoy\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.636785 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.636763    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-podinfo\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.653149 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.653090    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-token\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.653378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.653341    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csmg6\" (UniqueName: \"kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-kube-api-access-csmg6\") pod \"e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") " pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:50.710090 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:50.710058    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:51.022022 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:51.021987    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:51.030831 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:04:51.030802    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf382c2bb_0885_48f3_9ac1_09585a2b1c36.slice/crio-0e27f72a63e9bdce99f4d065ce3d34fca861e53d9d8bc41cbf1b20969b158d0a WatchSource:0}: Error finding container 0e27f72a63e9bdce99f4d065ce3d34fca861e53d9d8bc41cbf1b20969b158d0a: Status 404 returned error can't find the container with id 0e27f72a63e9bdce99f4d065ce3d34fca861e53d9d8bc41cbf1b20969b158d0a
Oct 30 13:04:51.032047 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:51.032025    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"]
Oct 30 13:04:51.033463 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:51.033420    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:04:51.033576 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:51.033495    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:04:51.033576 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:51.033544    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:04:51.193390 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:04:51.185681    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:04:51.401828 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:51.401736    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv" event={"ID":"f382c2bb-0885-48f3-9ac1-09585a2b1c36","Type":"ContainerStarted","Data":"576ca5cc5e9688700d553d47b2e7923296909152b625b7f0033cb576ca0f982d"}
Oct 30 13:04:51.401828 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:51.401796    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv" event={"ID":"f382c2bb-0885-48f3-9ac1-09585a2b1c36","Type":"ContainerStarted","Data":"0e27f72a63e9bdce99f4d065ce3d34fca861e53d9d8bc41cbf1b20969b158d0a"}
Oct 30 13:04:51.447336 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:51.447274    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv" podStartSLOduration=1.447257205 podStartE2EDuration="1.447257205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:51.445907143 +0000 UTC m=+1735.261233631" watchObservedRunningTime="2026-10-30 13:04:51.447257205 +0000 UTC m=+1735.262583696"
Oct 30 13:04:51.710916 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:51.710814    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:52.397595 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:52.397556    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:52.726183 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:52.726074    4522 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:04:52.726832 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:52.726788    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv" podUID="f382c2bb-0885-48f3-9ac1-09585a2b1c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:04:53.090390 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:53.077892    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:53.710931 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:53.710899    4522 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:04:53.711084 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:53.710965    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv" podUID="f382c2bb-0885-48f3-9ac1-09585a2b1c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:04:54.711415 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:54.711378    4522 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:04:54.711817 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:54.711431    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv" podUID="f382c2bb-0885-48f3-9ac1-09585a2b1c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:04:54.860517 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:54.860481    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:55.043244 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:55.043206    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:55.339452 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:55.339342    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:55.466622 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:55.466592    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:55.711080 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:55.710998    4522 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:04:55.711080 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:55.711066    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv" podUID="f382c2bb-0885-48f3-9ac1-09585a2b1c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:04:56.062901 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:56.062852    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:56.823016 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:56.822974    4522 patch_prober.go:29] interesting pod/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:04:56.823239 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:56.823037    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv" podUID="f382c2bb-0885-48f3-9ac1-09585a2b1c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:04:57.068612 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:57.068566    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:57.206990 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:57.206897    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:57.406470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:57.406416    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:57.549272 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:57.549229    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:57.551247 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:57.551218    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"]
Oct 30 13:04:57.560380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:57.560327    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"]
Oct 30 13:04:57.650746 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:57.650713    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:58.191067 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:58.191034    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:58.191581 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:58.191514    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:58.192844 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:58.192821    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:04:59.307979 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:04:59.307943    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:01.045395 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:01.045325    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:01.213073 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:01.213037    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:01.937075 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:01.937018    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:02.061727 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:02.061687    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:02.194932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:02.194816    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:03.387203 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:03.387168    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"]
Oct 30 13:05:03.489863 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:03.489833    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:03.605257 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:03.605226    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:04.769703 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:04.769666    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:04.920628 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:04.920588    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:05.832199 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:05.832162    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:07.621887 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:07.621852    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:08.387671 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:08.387615    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf" podUID="b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878" containerName="payload-processing" containerID="cri-o://4013df4a250d1bfaafba6e11f197c44b137b189749337352107e8389d1f5f42c" gracePeriod=25
Oct 30 13:05:09.315082 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:09.315039    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf" podUID="b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:05:09.374645 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:09.374608    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:11.297664 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:11.297610    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:11.954552 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:11.954520    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:12.625794 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:12.625741    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main" containerID="cri-o://c491c293b178a495a60bf678dff8a1270ad5f7017b9cee444c4f815b817ea4c4" gracePeriod=45
Oct 30 13:05:12.641785 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:12.641749    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main" containerID="cri-o://dafd011a73ad621a1c513d23306b5aec503ee21fa493b0c27e4ed383fef1b42b" gracePeriod=45
Oct 30 13:05:12.742559 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:12.742529    4522 generic.go:356] "Generic (PLEG): container finished" podID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerID="c491c293b178a495a60bf678dff8a1270ad5f7017b9cee444c4f815b817ea4c4" exitCode=0
Oct 30 13:05:12.742706 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:12.742600    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" event={"ID":"f3fda1a0-5841-4eaa-9621-1fe8b5165b12","Type":"ContainerDied","Data":"c491c293b178a495a60bf678dff8a1270ad5f7017b9cee444c4f815b817ea4c4"}
Oct 30 13:05:13.073814 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.073513    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:05:13.093501 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.093473    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:05:13.167464 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.167344    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03285671-c2a3-4503-accb-e6e3b41a16a5-kube-api-access-9dpdp\" (UniqueName: \"kubernetes.io/projected/03285671-c2a3-4503-accb-e6e3b41a16a5-kube-api-access-9dpdp\") pod \"03285671-c2a3-4503-accb-e6e3b41a16a5\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") "
Oct 30 13:05:13.167464 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.167440    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tls-certs\") pod \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") "
Oct 30 13:05:13.167689 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.167504    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-kube-api-access-x27xr\" (UniqueName: \"kubernetes.io/projected/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-kube-api-access-x27xr\") pod \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") "
Oct 30 13:05:13.167689 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.167540    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-home\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-home\") pod \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") "
Oct 30 13:05:13.167689 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.167643    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03285671-c2a3-4503-accb-e6e3b41a16a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/03285671-c2a3-4503-accb-e6e3b41a16a5-tls-certs\") pod \"03285671-c2a3-4503-accb-e6e3b41a16a5\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") "
Oct 30 13:05:13.167873 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.167716    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-dshm\") pod \"03285671-c2a3-4503-accb-e6e3b41a16a5\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") "
Oct 30 13:05:13.167873 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.167759    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-home\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-home\") pod \"03285671-c2a3-4503-accb-e6e3b41a16a5\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") "
Oct 30 13:05:13.167873 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.167802    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-model-cache\") pod \"03285671-c2a3-4503-accb-e6e3b41a16a5\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") "
Oct 30 13:05:13.167873 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.167848    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-model-cache\") pod \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") "
Oct 30 13:05:13.168094 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.167909    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-tmp-dir\") pod \"03285671-c2a3-4503-accb-e6e3b41a16a5\" (UID: \"03285671-c2a3-4503-accb-e6e3b41a16a5\") "
Oct 30 13:05:13.168094 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.167933    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-dshm\") pod \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") "
Oct 30 13:05:13.168199 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.168100    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tmp-dir\") pod \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\" (UID: \"f3fda1a0-5841-4eaa-9621-1fe8b5165b12\") "
Oct 30 13:05:13.168605 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.168576    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-home" pod "03285671-c2a3-4503-accb-e6e3b41a16a5" (UID: "03285671-c2a3-4503-accb-e6e3b41a16a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:13.169563 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.168864    4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-home\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:13.169563 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.168949    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-home" pod "f3fda1a0-5841-4eaa-9621-1fe8b5165b12" (UID: "f3fda1a0-5841-4eaa-9621-1fe8b5165b12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:13.169563 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.169029    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-model-cache" pod "f3fda1a0-5841-4eaa-9621-1fe8b5165b12" (UID: "f3fda1a0-5841-4eaa-9621-1fe8b5165b12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:13.169563 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.169139    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-tmp-dir" pod "03285671-c2a3-4503-accb-e6e3b41a16a5" (UID: "03285671-c2a3-4503-accb-e6e3b41a16a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:13.169563 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.169255    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-model-cache" pod "03285671-c2a3-4503-accb-e6e3b41a16a5" (UID: "03285671-c2a3-4503-accb-e6e3b41a16a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:13.169563 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.169338    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tmp-dir" pod "f3fda1a0-5841-4eaa-9621-1fe8b5165b12" (UID: "f3fda1a0-5841-4eaa-9621-1fe8b5165b12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:13.173025 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.172989    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-kube-api-access-x27xr" pod "f3fda1a0-5841-4eaa-9621-1fe8b5165b12" (UID: "f3fda1a0-5841-4eaa-9621-1fe8b5165b12"). InnerVolumeSpecName "kube-api-access-x27xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:13.174013 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.173982    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-dshm" pod "f3fda1a0-5841-4eaa-9621-1fe8b5165b12" (UID: "f3fda1a0-5841-4eaa-9621-1fe8b5165b12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:13.174114 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.174082    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tls-certs" pod "f3fda1a0-5841-4eaa-9621-1fe8b5165b12" (UID: "f3fda1a0-5841-4eaa-9621-1fe8b5165b12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:13.174182 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.174138    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03285671-c2a3-4503-accb-e6e3b41a16a5-tls-certs" pod "03285671-c2a3-4503-accb-e6e3b41a16a5" (UID: "03285671-c2a3-4503-accb-e6e3b41a16a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:13.174693 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.174660    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-dshm" pod "03285671-c2a3-4503-accb-e6e3b41a16a5" (UID: "03285671-c2a3-4503-accb-e6e3b41a16a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:13.175391 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.175332    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03285671-c2a3-4503-accb-e6e3b41a16a5-kube-api-access-9dpdp" pod "03285671-c2a3-4503-accb-e6e3b41a16a5" (UID: "03285671-c2a3-4503-accb-e6e3b41a16a5"). InnerVolumeSpecName "kube-api-access-9dpdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:13.270929 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.270885    4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-dshm\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:13.270929 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.270931    4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-model-cache\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:13.271654 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.270946    4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-model-cache\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:13.271654 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.270960    4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03285671-c2a3-4503-accb-e6e3b41a16a5-tmp-dir\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:13.271654 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.270974    4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-dshm\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:13.271654 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.270987    4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tmp-dir\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:13.271654 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.271016    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dpdp\" (UniqueName: \"kubernetes.io/projected/03285671-c2a3-4503-accb-e6e3b41a16a5-kube-api-access-9dpdp\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:13.271654 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.271029    4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-tls-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:13.271654 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.271044    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x27xr\" (UniqueName: \"kubernetes.io/projected/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-kube-api-access-x27xr\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:13.271654 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.271060    4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3fda1a0-5841-4eaa-9621-1fe8b5165b12-home\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:13.271654 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.271073    4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03285671-c2a3-4503-accb-e6e3b41a16a5-tls-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:13.750075 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.750043    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"
Oct 30 13:05:13.750533 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.750049    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw" event={"ID":"f3fda1a0-5841-4eaa-9621-1fe8b5165b12","Type":"ContainerDied","Data":"417e93e9f078c3047bc1ec3e538c2f4f98aa66e9fa29e1bd5351d3d2a2cac394"}
Oct 30 13:05:13.750533 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.750177    4522 scope.go:122] "RemoveContainer" containerID="c491c293b178a495a60bf678dff8a1270ad5f7017b9cee444c4f815b817ea4c4"
Oct 30 13:05:13.751628 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.751512    4522 generic.go:356] "Generic (PLEG): container finished" podID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerID="dafd011a73ad621a1c513d23306b5aec503ee21fa493b0c27e4ed383fef1b42b" exitCode=0
Oct 30 13:05:13.751628 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.751548    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" event={"ID":"03285671-c2a3-4503-accb-e6e3b41a16a5","Type":"ContainerDied","Data":"dafd011a73ad621a1c513d23306b5aec503ee21fa493b0c27e4ed383fef1b42b"}
Oct 30 13:05:13.751628 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.751559    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"
Oct 30 13:05:13.751628 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.751574    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf" event={"ID":"03285671-c2a3-4503-accb-e6e3b41a16a5","Type":"ContainerDied","Data":"2f522986ffe064e3bb0ab3f378ec22d3cecac422a31f235da721be8c95168545"}
Oct 30 13:05:13.760482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.760461    4522 scope.go:122] "RemoveContainer" containerID="dafd011a73ad621a1c513d23306b5aec503ee21fa493b0c27e4ed383fef1b42b"
Oct 30 13:05:13.769008 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.768990    4522 scope.go:122] "RemoveContainer" containerID="dafd011a73ad621a1c513d23306b5aec503ee21fa493b0c27e4ed383fef1b42b"
Oct 30 13:05:13.769258 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:05:13.769239    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafd011a73ad621a1c513d23306b5aec503ee21fa493b0c27e4ed383fef1b42b\": container with ID starting with dafd011a73ad621a1c513d23306b5aec503ee21fa493b0c27e4ed383fef1b42b not found: ID does not exist" containerID="dafd011a73ad621a1c513d23306b5aec503ee21fa493b0c27e4ed383fef1b42b"
Oct 30 13:05:13.769306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.769265    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafd011a73ad621a1c513d23306b5aec503ee21fa493b0c27e4ed383fef1b42b"} err="failed to get container status \"dafd011a73ad621a1c513d23306b5aec503ee21fa493b0c27e4ed383fef1b42b\": rpc error: code = NotFound desc = could not find container \"dafd011a73ad621a1c513d23306b5aec503ee21fa493b0c27e4ed383fef1b42b\": container with ID starting with dafd011a73ad621a1c513d23306b5aec503ee21fa493b0c27e4ed383fef1b42b not found: ID does not exist"
Oct 30 13:05:13.808913 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.808892    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"]
Oct 30 13:05:13.820183 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.820162    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-105228/simulator-w2-105228-kserve-66d756bcfc-7kfnf"]
Oct 30 13:05:13.873705 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.873679    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"]
Oct 30 13:05:13.885299 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:13.885274    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-105228/premium-w2-105228-kserve-7bf6cb7f6d-bwvqw"]
Oct 30 13:05:14.047189 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:14.047050    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:14.475421 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:14.474798    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" path="/var/lib/kubelet/pods/03285671-c2a3-4503-accb-e6e3b41a16a5/volumes"
Oct 30 13:05:14.475604 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:14.475565    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" path="/var/lib/kubelet/pods/f3fda1a0-5841-4eaa-9621-1fe8b5165b12/volumes"
Oct 30 13:05:14.715422 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:14.714366    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:14.816826 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:14.816791    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:14.918075 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:14.918041    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:15.343471 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:15.343436    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:16.864531 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:16.864499    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:18.721732 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:18.721691    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:19.315066 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:19.315031    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf" podUID="b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:05:20.399115 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:20.399082    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:21.137785 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:05:21.137745    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:05:21.652165 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:21.652130    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:21.809608 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:21.809574    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:22.046325 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:22.046293    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:22.403033 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:22.402945    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:24.360842 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:24.360812    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:24.484196 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:24.484156    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:24.741417 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:24.741327    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:26.408571 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:26.408509    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:28.181504 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.181462    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:28.493031 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.492960    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"]
Oct 30 13:05:28.493188 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.493154    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx" podUID="b68996c3-3081-47c8-9878-d2e438a298a7" containerName="discovery" containerID="cri-o://9465fccc24422763da2a1c7e03c8e8f8be411b6b7c10d5069c0595361263c6c1" gracePeriod=30
Oct 30 13:05:28.698077 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.698048    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"
Oct 30 13:05:28.758755 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.758654    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-plugins-config-volume\") pod \"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878\" (UID: \"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878\") "
Oct 30 13:05:28.758923 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.758794    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-kube-api-access-8g495\" (UniqueName: \"kubernetes.io/projected/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-kube-api-access-8g495\") pod \"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878\" (UID: \"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878\") "
Oct 30 13:05:28.759450 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.759419    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-plugins-config-volume" pod "b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878" (UID: "b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:28.765239 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.765211    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-kube-api-access-8g495" pod "b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878" (UID: "b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878"). InnerVolumeSpecName "kube-api-access-8g495". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:28.802477 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.802453    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 13:05:28.873374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.861598    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68996c3-3081-47c8-9878-d2e438a298a7-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68996c3-3081-47c8-9878-d2e438a298a7-local-certs\") pod \"b68996c3-3081-47c8-9878-d2e438a298a7\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") "
Oct 30 13:05:28.873374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.861688    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-kube-api-access-5mc4j\" (UniqueName: \"kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-kube-api-access-5mc4j\") pod \"b68996c3-3081-47c8-9878-d2e438a298a7\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") "
Oct 30 13:05:28.873374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.861716    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-kubeconfig\") pod \"b68996c3-3081-47c8-9878-d2e438a298a7\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") "
Oct 30 13:05:28.873374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.861776    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-ca-configmap\") pod \"b68996c3-3081-47c8-9878-d2e438a298a7\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") "
Oct 30 13:05:28.873374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.861805    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-dns-cert\") pod \"b68996c3-3081-47c8-9878-d2e438a298a7\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") "
Oct 30 13:05:28.873374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.861856    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-cacerts\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-cacerts\") pod \"b68996c3-3081-47c8-9878-d2e438a298a7\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") "
Oct 30 13:05:28.873374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.861916    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-istio-token\" (UniqueName: \"kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-istio-token\") pod \"b68996c3-3081-47c8-9878-d2e438a298a7\" (UID: \"b68996c3-3081-47c8-9878-d2e438a298a7\") "
Oct 30 13:05:28.873374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.862551    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:28.873374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.862571    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g495\" (UniqueName: \"kubernetes.io/projected/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878-kube-api-access-8g495\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:28.880764 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.879248    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-ca-configmap" pod "b68996c3-3081-47c8-9878-d2e438a298a7" (UID: "b68996c3-3081-47c8-9878-d2e438a298a7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:28.881535 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.881411    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-istio-token" pod "b68996c3-3081-47c8-9878-d2e438a298a7" (UID: "b68996c3-3081-47c8-9878-d2e438a298a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:28.881535 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.881425    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-cacerts" pod "b68996c3-3081-47c8-9878-d2e438a298a7" (UID: "b68996c3-3081-47c8-9878-d2e438a298a7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:28.881535 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.881496    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68996c3-3081-47c8-9878-d2e438a298a7-local-certs" pod "b68996c3-3081-47c8-9878-d2e438a298a7" (UID: "b68996c3-3081-47c8-9878-d2e438a298a7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:28.881795 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.881548    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-dns-cert" pod "b68996c3-3081-47c8-9878-d2e438a298a7" (UID: "b68996c3-3081-47c8-9878-d2e438a298a7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:28.882993 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.882972    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-kubeconfig" pod "b68996c3-3081-47c8-9878-d2e438a298a7" (UID: "b68996c3-3081-47c8-9878-d2e438a298a7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:28.884755 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.884724    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-kube-api-access-5mc4j" pod "b68996c3-3081-47c8-9878-d2e438a298a7" (UID: "b68996c3-3081-47c8-9878-d2e438a298a7"). InnerVolumeSpecName "kube-api-access-5mc4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:28.910991 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.910964    4522 generic.go:356] "Generic (PLEG): container finished" podID="b68996c3-3081-47c8-9878-d2e438a298a7" containerID="9465fccc24422763da2a1c7e03c8e8f8be411b6b7c10d5069c0595361263c6c1" exitCode=0
Oct 30 13:05:28.911093 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.911015    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"
Oct 30 13:05:28.911093 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.911049    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx" event={"ID":"b68996c3-3081-47c8-9878-d2e438a298a7","Type":"ContainerDied","Data":"9465fccc24422763da2a1c7e03c8e8f8be411b6b7c10d5069c0595361263c6c1"}
Oct 30 13:05:28.911182 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.911102    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx" event={"ID":"b68996c3-3081-47c8-9878-d2e438a298a7","Type":"ContainerDied","Data":"262e594f140924e795250beea43a377762152d7bd0ae788c7ec987715bb0e353"}
Oct 30 13:05:28.911182 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.911123    4522 scope.go:122] "RemoveContainer" containerID="9465fccc24422763da2a1c7e03c8e8f8be411b6b7c10d5069c0595361263c6c1"
Oct 30 13:05:28.912523 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.912499    4522 generic.go:356] "Generic (PLEG): container finished" podID="b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878" containerID="4013df4a250d1bfaafba6e11f197c44b137b189749337352107e8389d1f5f42c" exitCode=0
Oct 30 13:05:28.912606 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.912546    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"
Oct 30 13:05:28.912606 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.912589    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf" event={"ID":"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878","Type":"ContainerDied","Data":"4013df4a250d1bfaafba6e11f197c44b137b189749337352107e8389d1f5f42c"}
Oct 30 13:05:28.912718 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.912640    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf" event={"ID":"b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878","Type":"ContainerDied","Data":"4b00298e2b4b135a19064805bf51d778c6c7ad35dd0a5219a330f33fc02c96a8"}
Oct 30 13:05:28.921187 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.921168    4522 scope.go:122] "RemoveContainer" containerID="9465fccc24422763da2a1c7e03c8e8f8be411b6b7c10d5069c0595361263c6c1"
Oct 30 13:05:28.921459 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:05:28.921439    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9465fccc24422763da2a1c7e03c8e8f8be411b6b7c10d5069c0595361263c6c1\": container with ID starting with 9465fccc24422763da2a1c7e03c8e8f8be411b6b7c10d5069c0595361263c6c1 not found: ID does not exist" containerID="9465fccc24422763da2a1c7e03c8e8f8be411b6b7c10d5069c0595361263c6c1"
Oct 30 13:05:28.921539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.921465    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9465fccc24422763da2a1c7e03c8e8f8be411b6b7c10d5069c0595361263c6c1"} err="failed to get container status \"9465fccc24422763da2a1c7e03c8e8f8be411b6b7c10d5069c0595361263c6c1\": rpc error: code = NotFound desc = could not find container \"9465fccc24422763da2a1c7e03c8e8f8be411b6b7c10d5069c0595361263c6c1\": container with ID starting with 9465fccc24422763da2a1c7e03c8e8f8be411b6b7c10d5069c0595361263c6c1 not found: ID does not exist"
Oct 30 13:05:28.921539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.921480    4522 scope.go:122] "RemoveContainer" containerID="4013df4a250d1bfaafba6e11f197c44b137b189749337352107e8389d1f5f42c"
Oct 30 13:05:28.932734 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.932717    4522 scope.go:122] "RemoveContainer" containerID="4013df4a250d1bfaafba6e11f197c44b137b189749337352107e8389d1f5f42c"
Oct 30 13:05:28.933046 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:05:28.933027    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4013df4a250d1bfaafba6e11f197c44b137b189749337352107e8389d1f5f42c\": container with ID starting with 4013df4a250d1bfaafba6e11f197c44b137b189749337352107e8389d1f5f42c not found: ID does not exist" containerID="4013df4a250d1bfaafba6e11f197c44b137b189749337352107e8389d1f5f42c"
Oct 30 13:05:28.933125 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.933054    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4013df4a250d1bfaafba6e11f197c44b137b189749337352107e8389d1f5f42c"} err="failed to get container status \"4013df4a250d1bfaafba6e11f197c44b137b189749337352107e8389d1f5f42c\": rpc error: code = NotFound desc = could not find container \"4013df4a250d1bfaafba6e11f197c44b137b189749337352107e8389d1f5f42c\": container with ID starting with 4013df4a250d1bfaafba6e11f197c44b137b189749337352107e8389d1f5f42c not found: ID does not exist"
Oct 30 13:05:28.964845 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.964497    4522 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68996c3-3081-47c8-9878-d2e438a298a7-local-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:28.964845 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.964527    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mc4j\" (UniqueName: \"kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-kube-api-access-5mc4j\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:28.964845 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.964620    4522 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-kubeconfig\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:28.965074 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.964999    4522 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-ca-configmap\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:28.965074 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.965020    4522 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-istio-csr-dns-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:28.965074 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.965033    4522 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b68996c3-3081-47c8-9878-d2e438a298a7-cacerts\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:28.965074 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.965048    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68996c3-3081-47c8-9878-d2e438a298a7-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:28.965762 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.965692    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"]
Oct 30 13:05:28.978827 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:28.978787    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gjccx"]
Oct 30 13:05:29.014748 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:29.014667    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"]
Oct 30 13:05:29.017746 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:29.017723    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-105228-554d9ddf7d-zfvmf"]
Oct 30 13:05:29.985333 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:29.985287    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:30.334461 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:30.334427    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878" path="/var/lib/kubelet/pods/b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878/volumes"
Oct 30 13:05:30.334927 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:30.334912    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68996c3-3081-47c8-9878-d2e438a298a7" path="/var/lib/kubelet/pods/b68996c3-3081-47c8-9878-d2e438a298a7/volumes"
Oct 30 13:05:31.759951 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:31.759919    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:33.120408 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:33.120347    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:33.242588 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:33.242549    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:33.564681 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:33.564645    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:34.421481 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:34.421448    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:34.545023 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:34.544983    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:35.746654 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:35.746623    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"]
Oct 30 13:05:35.747159 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:35.746836    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5" podUID="753b333f-cc81-4df9-ad51-e14445ce8fa4" containerName="maas-api" containerID="cri-o://246e8f45056bee3ac666d1fdfcf150746af43f9f22bed74e1a8386554b349541" gracePeriod=30
Oct 30 13:05:35.817103 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:35.817071    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:36.385850 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:36.385802    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5" podUID="753b333f-cc81-4df9-ad51-e14445ce8fa4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.56:8443/health\": dial tcp 10.133.0.56:8443: connect: connection refused"
Oct 30 13:05:36.454517 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:36.454070    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:36.571378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:36.556429    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:36.663608 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:36.662399    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:36.824219 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:36.824186    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:37.006146 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.005115    4522 generic.go:356] "Generic (PLEG): container finished" podID="753b333f-cc81-4df9-ad51-e14445ce8fa4" containerID="246e8f45056bee3ac666d1fdfcf150746af43f9f22bed74e1a8386554b349541" exitCode=0
Oct 30 13:05:37.006146 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.005189    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5" event={"ID":"753b333f-cc81-4df9-ad51-e14445ce8fa4","Type":"ContainerDied","Data":"246e8f45056bee3ac666d1fdfcf150746af43f9f22bed74e1a8386554b349541"}
Oct 30 13:05:37.124287 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.124261    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 13:05:37.175257 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.175226    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:37.220455 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.220415    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-maas-api-tls\") pod \"753b333f-cc81-4df9-ad51-e14445ce8fa4\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") "
Oct 30 13:05:37.220639 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.220540    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/753b333f-cc81-4df9-ad51-e14445ce8fa4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/753b333f-cc81-4df9-ad51-e14445ce8fa4-openshift-service-ca\") pod \"753b333f-cc81-4df9-ad51-e14445ce8fa4\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") "
Oct 30 13:05:37.220778 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.220757    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-metrics-certs\") pod \"753b333f-cc81-4df9-ad51-e14445ce8fa4\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") "
Oct 30 13:05:37.220858 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.220815    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/753b333f-cc81-4df9-ad51-e14445ce8fa4-kube-api-access-sbl5q\" (UniqueName: \"kubernetes.io/projected/753b333f-cc81-4df9-ad51-e14445ce8fa4-kube-api-access-sbl5q\") pod \"753b333f-cc81-4df9-ad51-e14445ce8fa4\" (UID: \"753b333f-cc81-4df9-ad51-e14445ce8fa4\") "
Oct 30 13:05:37.220914 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.220887    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/753b333f-cc81-4df9-ad51-e14445ce8fa4-openshift-service-ca" pod "753b333f-cc81-4df9-ad51-e14445ce8fa4" (UID: "753b333f-cc81-4df9-ad51-e14445ce8fa4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:37.221605 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.221579    4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/753b333f-cc81-4df9-ad51-e14445ce8fa4-openshift-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:37.223236 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.223197    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753b333f-cc81-4df9-ad51-e14445ce8fa4-kube-api-access-sbl5q" pod "753b333f-cc81-4df9-ad51-e14445ce8fa4" (UID: "753b333f-cc81-4df9-ad51-e14445ce8fa4"). InnerVolumeSpecName "kube-api-access-sbl5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:37.223338 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.223299    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-maas-api-tls" pod "753b333f-cc81-4df9-ad51-e14445ce8fa4" (UID: "753b333f-cc81-4df9-ad51-e14445ce8fa4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:37.223441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.223402    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-metrics-certs" pod "753b333f-cc81-4df9-ad51-e14445ce8fa4" (UID: "753b333f-cc81-4df9-ad51-e14445ce8fa4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:37.325432 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.325400    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:37.325432 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.325430    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbl5q\" (UniqueName: \"kubernetes.io/projected/753b333f-cc81-4df9-ad51-e14445ce8fa4-kube-api-access-sbl5q\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:37.325432 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:37.325439    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/753b333f-cc81-4df9-ad51-e14445ce8fa4-maas-api-tls\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:38.014034 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:38.014006    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"
Oct 30 13:05:38.014471 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:38.014035    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5" event={"ID":"753b333f-cc81-4df9-ad51-e14445ce8fa4","Type":"ContainerDied","Data":"bbf9f1a89836a97aa010eb2a7a823b2ef78161d634b585f22707a833db34fb97"}
Oct 30 13:05:38.014471 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:38.014094    4522 scope.go:122] "RemoveContainer" containerID="246e8f45056bee3ac666d1fdfcf150746af43f9f22bed74e1a8386554b349541"
Oct 30 13:05:38.067493 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:38.067466    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"]
Oct 30 13:05:38.077876 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:38.077847    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-105228-cfbbbc484-rg2v5"]
Oct 30 13:05:38.334448 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:38.334421    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="753b333f-cc81-4df9-ad51-e14445ce8fa4" path="/var/lib/kubelet/pods/753b333f-cc81-4df9-ad51-e14445ce8fa4/volumes"
Oct 30 13:05:39.259555 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:39.259517    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:40.961135 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:40.961100    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:42.627228 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:42.627181    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:42.985287 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:42.985207    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:43.128161 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:43.128118    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:43.437747 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:43.437712    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:43.557376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:43.554946    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:43.871459 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:43.871428    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"]
Oct 30 13:05:43.871935 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:43.871702    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy" containerID="cri-o://2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f" gracePeriod=30
Oct 30 13:05:43.884646 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:43.884622    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"]
Oct 30 13:05:43.884835 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:43.884809    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" containerID="cri-o://4e45cdbd858a3ff5675c2dae4a6c12621fc1d1af7b42f4700eb8aee7b45a9f00" gracePeriod=30
Oct 30 13:05:43.895963 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:43.895938    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:43.896973 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:43.896952    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"]
Oct 30 13:05:43.897145 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:43.897114    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy" containerID="cri-o://72626aae9e9e3a957ae9125660854ec173922e4c29576a7cf007e08f2f7f87ff" gracePeriod=30
Oct 30 13:05:43.910208 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:43.910184    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"]
Oct 30 13:05:43.910412 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:43.910389    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy" containerID="cri-o://47f88fd4d522d009d64c4a880d593e7578975e00ed156b3e934a294a32a4a3a9" gracePeriod=30
Oct 30 13:05:44.636837 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:44.636798    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:46.021442 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:46.003934    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 13:05:46.384960 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:46.384877    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:47.186390 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:47.186341    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:47.428394 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:47.428332    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:47.960512 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:47.960433    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:49.357820 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.357469    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 13:05:49.357820 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.357545    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 13:05:49.412301 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.411737    4522 generic.go:356] "Generic (PLEG): container finished" podID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerID="72626aae9e9e3a957ae9125660854ec173922e4c29576a7cf007e08f2f7f87ff" exitCode=0
Oct 30 13:05:49.412301 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.411868    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" event={"ID":"b45b4a4c-94e4-4897-8523-0e446e994bea","Type":"ContainerDied","Data":"72626aae9e9e3a957ae9125660854ec173922e4c29576a7cf007e08f2f7f87ff"}
Oct 30 13:05:49.490445 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.466736    4522 generic.go:356] "Generic (PLEG): container finished" podID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerID="4e45cdbd858a3ff5675c2dae4a6c12621fc1d1af7b42f4700eb8aee7b45a9f00" exitCode=0
Oct 30 13:05:49.490445 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.466834    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" event={"ID":"4e6f2bb2-7f4c-4672-a201-9b822bea3687","Type":"ContainerDied","Data":"4e45cdbd858a3ff5675c2dae4a6c12621fc1d1af7b42f4700eb8aee7b45a9f00"}
Oct 30 13:05:49.490445 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.469644    4522 generic.go:356] "Generic (PLEG): container finished" podID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerID="2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f" exitCode=0
Oct 30 13:05:49.490445 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.469735    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" event={"ID":"8e76d3f0-c1f4-4458-be8d-8d6922001217","Type":"ContainerDied","Data":"2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f"}
Oct 30 13:05:49.490445 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.469780    4522 scope.go:122] "RemoveContainer" containerID="d61531e53dc65dbe5434cb28f9413bd502ceff2b7e1cd96f0a8e2a9166657fd6"
Oct 30 13:05:49.555387 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.544514    4522 generic.go:356] "Generic (PLEG): container finished" podID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerID="47f88fd4d522d009d64c4a880d593e7578975e00ed156b3e934a294a32a4a3a9" exitCode=0
Oct 30 13:05:49.555387 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.544585    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" event={"ID":"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef","Type":"ContainerDied","Data":"47f88fd4d522d009d64c4a880d593e7578975e00ed156b3e934a294a32a4a3a9"}
Oct 30 13:05:49.896545 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.896508    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:49.935457 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.935424    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 13:05:49.935736 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:49.935710    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 13:05:50.042386 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:50.041461    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:50.042386 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:50.041884    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:05:50.099199 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:50.099129    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:05:50.133385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:50.128474    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:50.349996 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:50.349951    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:50.441556 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:50.440451    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:50.467479 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:50.467436    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:50.467921 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:50.467888    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:05:50.641703 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:50.641576    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:50.708180 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:50.708144    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:50.954681 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:50.954589    4522 patch_prober.go:29] interesting pod/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:05:50.958375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:50.954917    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:05:51.049501 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.049448    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:05:51.084377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.050954    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:05:51.085626 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.070429    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:05:51.085626 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.084767    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:05:51.325389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.311694    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:51.364383 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.353426    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:05:51.389381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.364761    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:05:51.389381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.365562    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:05:51.389381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.365820    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 30 13:05:51.389381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.365939    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:05:51.389381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.365998    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:05:51.462376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.459441    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:51.462376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.459876    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:05:51.484281 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.483927    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:51.499402 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.494826    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:05:51.526230 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.521443    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded"
Oct 30 13:05:51.622437 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.593914    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:51.842523 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:51.842474    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:52.080383 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:52.075277    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:52.100533 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:52.100481    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:05:52.543797 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:52.543748    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:05:52.986582 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:52.986484    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:05:53.040389 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:05:53.000786    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:05:53.040389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.000851    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:05:53.040389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.002722    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:53.041065 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.002753    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-z7swk" podUID="29d90fc8-e584-45cb-9e65-dfaa8c7bc226" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:53.050020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.027433    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:53.060036 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.028024    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:53.148381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.072666    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj" podUID="9f133540-b63f-449a-8377-36912a0f1b35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 13:05:53.148381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.028042    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:05:53.148381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.099184    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:53.148381 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.120848    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:53.241637 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.183014    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:53.344477 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.329508    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 13:05:53.414378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.413559    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istiod-ca-cert" pod "4e6f2bb2-7f4c-4672-a201-9b822bea3687" (UID: "4e6f2bb2-7f4c-4672-a201-9b822bea3687"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:53.434808 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.432060    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:53.434808 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.432434    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.512618    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istiod-ca-cert\") pod \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") "
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.512732    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-envoy\") pod \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") "
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.512827    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-token\" (UniqueName: \"kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-token\") pod \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") "
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.512877    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-credential-socket\") pod \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") "
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.512938    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-podinfo\") pod \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") "
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.513057    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-certs\") pod \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") "
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.513098    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-data\") pod \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") "
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.513302    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-kube-api-access-zcwmt\" (UniqueName: \"kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-kube-api-access-zcwmt\") pod \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") "
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.513414    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-socket\") pod \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\" (UID: \"4e6f2bb2-7f4c-4672-a201-9b822bea3687\") "
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.514221    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.540258    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-data" pod "4e6f2bb2-7f4c-4672-a201-9b822bea3687" (UID: "4e6f2bb2-7f4c-4672-a201-9b822bea3687"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.540546    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-certs" pod "4e6f2bb2-7f4c-4672-a201-9b822bea3687" (UID: "4e6f2bb2-7f4c-4672-a201-9b822bea3687"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.540961    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-socket" pod "4e6f2bb2-7f4c-4672-a201-9b822bea3687" (UID: "4e6f2bb2-7f4c-4672-a201-9b822bea3687"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.606687 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.550528    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:53.690678 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.658997    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-envoy" pod "4e6f2bb2-7f4c-4672-a201-9b822bea3687" (UID: "4e6f2bb2-7f4c-4672-a201-9b822bea3687"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.690678 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.660928    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-kube-api-access-zcwmt" pod "4e6f2bb2-7f4c-4672-a201-9b822bea3687" (UID: "4e6f2bb2-7f4c-4672-a201-9b822bea3687"). InnerVolumeSpecName "kube-api-access-zcwmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:53.690678 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.661008    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-token" pod "4e6f2bb2-7f4c-4672-a201-9b822bea3687" (UID: "4e6f2bb2-7f4c-4672-a201-9b822bea3687"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:53.690678 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.683556    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 13:05:53.744319 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.743561    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 13:05:53.918306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.817864    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 13:05:53.918306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.828599    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-credential-socket" pod "4e6f2bb2-7f4c-4672-a201-9b822bea3687" (UID: "4e6f2bb2-7f4c-4672-a201-9b822bea3687"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:53.918306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.829832    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-podinfo" pod "4e6f2bb2-7f4c-4672-a201-9b822bea3687" (UID: "4e6f2bb2-7f4c-4672-a201-9b822bea3687"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:05:53.918306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.915690    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.918306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.915913    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.918306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.915935    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.918306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.916147    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.918306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.916166    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.918306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.916180    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.918306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.916193    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6f2bb2-7f4c-4672-a201-9b822bea3687-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:53.918306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:53.916383    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcwmt\" (UniqueName: \"kubernetes.io/projected/4e6f2bb2-7f4c-4672-a201-9b822bea3687-kube-api-access-zcwmt\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.024824    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45b4a4c-94e4-4897-8523-0e446e994bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45b4a4c-94e4-4897-8523-0e446e994bea-istiod-ca-cert\") pod \"b45b4a4c-94e4-4897-8523-0e446e994bea\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.024882    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-credential-socket\") pod \"8e76d3f0-c1f4-4458-be8d-8d6922001217\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.024952    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-certs\") pod \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025040    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-podinfo\") pod \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025072    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-socket\") pod \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025124    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e76d3f0-c1f4-4458-be8d-8d6922001217-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e76d3f0-c1f4-4458-be8d-8d6922001217-istiod-ca-cert\") pod \"8e76d3f0-c1f4-4458-be8d-8d6922001217\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025167    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-envoy\") pod \"8e76d3f0-c1f4-4458-be8d-8d6922001217\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025201    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-token\" (UniqueName: \"kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-token\") pod \"8e76d3f0-c1f4-4458-be8d-8d6922001217\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025235    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-podinfo\") pod \"b45b4a4c-94e4-4897-8523-0e446e994bea\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025282    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-envoy\") pod \"b45b4a4c-94e4-4897-8523-0e446e994bea\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025339    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-token\" (UniqueName: \"kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-token\") pod \"b45b4a4c-94e4-4897-8523-0e446e994bea\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025411    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-credential-socket\") pod \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025462    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-data\") pod \"b45b4a4c-94e4-4897-8523-0e446e994bea\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025523    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istiod-ca-cert\") pod \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025556    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-podinfo\") pod \"8e76d3f0-c1f4-4458-be8d-8d6922001217\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025588    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-data\") pod \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025638    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-kube-api-access-dxq6d\" (UniqueName: \"kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-kube-api-access-dxq6d\") pod \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025702    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-certs\") pod \"8e76d3f0-c1f4-4458-be8d-8d6922001217\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025754    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-certs\") pod \"b45b4a4c-94e4-4897-8523-0e446e994bea\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025780    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-token\" (UniqueName: \"kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-token\") pod \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025835    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-socket\") pod \"b45b4a4c-94e4-4897-8523-0e446e994bea\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025873    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-envoy\") pod \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\" (UID: \"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025906    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-kube-api-access-rdv8q\" (UniqueName: \"kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-kube-api-access-rdv8q\") pod \"b45b4a4c-94e4-4897-8523-0e446e994bea\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025948    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-credential-socket\") pod \"b45b4a4c-94e4-4897-8523-0e446e994bea\" (UID: \"b45b4a4c-94e4-4897-8523-0e446e994bea\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.025997    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-kube-api-access-sd7w7\" (UniqueName: \"kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-kube-api-access-sd7w7\") pod \"8e76d3f0-c1f4-4458-be8d-8d6922001217\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.026030    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-socket\") pod \"8e76d3f0-c1f4-4458-be8d-8d6922001217\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") "
Oct 30 13:05:54.034380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.026057    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-data\") pod \"8e76d3f0-c1f4-4458-be8d-8d6922001217\" (UID: \"8e76d3f0-c1f4-4458-be8d-8d6922001217\") "
Oct 30 13:05:54.061385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.046530    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:54.061385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.047020    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:54.061385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.057417    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:54.061385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.058058    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-podinfo" pod "b45b4a4c-94e4-4897-8523-0e446e994bea" (UID: "b45b4a4c-94e4-4897-8523-0e446e994bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:05:54.061385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.060792    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45b4a4c-94e4-4897-8523-0e446e994bea-istiod-ca-cert" pod "b45b4a4c-94e4-4897-8523-0e446e994bea" (UID: "b45b4a4c-94e4-4897-8523-0e446e994bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:54.061385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.061072    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-credential-socket" pod "8e76d3f0-c1f4-4458-be8d-8d6922001217" (UID: "8e76d3f0-c1f4-4458-be8d-8d6922001217"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.061385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.061263    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-certs" pod "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" (UID: "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.062705 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.062448    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-envoy" pod "b45b4a4c-94e4-4897-8523-0e446e994bea" (UID: "b45b4a4c-94e4-4897-8523-0e446e994bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.067792 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.065740    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:54.069469 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.069420    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-data" pod "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" (UID: "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.084187 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.083430    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.073366    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-kube-api-access-dxq6d" pod "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" (UID: "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef"). InnerVolumeSpecName "kube-api-access-dxq6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.073542    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-certs" pod "8e76d3f0-c1f4-4458-be8d-8d6922001217" (UID: "8e76d3f0-c1f4-4458-be8d-8d6922001217"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.073688    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-certs" pod "b45b4a4c-94e4-4897-8523-0e446e994bea" (UID: "b45b4a4c-94e4-4897-8523-0e446e994bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.075612    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-credential-socket" pod "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" (UID: "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.075775    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-data" pod "b45b4a4c-94e4-4897-8523-0e446e994bea" (UID: "b45b4a4c-94e4-4897-8523-0e446e994bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.076024    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istiod-ca-cert" pod "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" (UID: "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.077771    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-token" pod "b45b4a4c-94e4-4897-8523-0e446e994bea" (UID: "b45b4a4c-94e4-4897-8523-0e446e994bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.079470    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e76d3f0-c1f4-4458-be8d-8d6922001217-istiod-ca-cert" pod "8e76d3f0-c1f4-4458-be8d-8d6922001217" (UID: "8e76d3f0-c1f4-4458-be8d-8d6922001217"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.083946    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-socket" pod "b45b4a4c-94e4-4897-8523-0e446e994bea" (UID: "b45b4a4c-94e4-4897-8523-0e446e994bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.090455    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-podinfo" pod "8e76d3f0-c1f4-4458-be8d-8d6922001217" (UID: "8e76d3f0-c1f4-4458-be8d-8d6922001217"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.110575    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-token" pod "8e76d3f0-c1f4-4458-be8d-8d6922001217" (UID: "8e76d3f0-c1f4-4458-be8d-8d6922001217"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.116038    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-kube-api-access-rdv8q" pod "b45b4a4c-94e4-4897-8523-0e446e994bea" (UID: "b45b4a4c-94e4-4897-8523-0e446e994bea"). InnerVolumeSpecName "kube-api-access-rdv8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.133191    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-socket" pod "8e76d3f0-c1f4-4458-be8d-8d6922001217" (UID: "8e76d3f0-c1f4-4458-be8d-8d6922001217"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.133300    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-credential-socket" pod "b45b4a4c-94e4-4897-8523-0e446e994bea" (UID: "b45b4a4c-94e4-4897-8523-0e446e994bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.145454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.128680    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-socket" pod "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" (UID: "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.156711 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.156677    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-envoy" pod "8e76d3f0-c1f4-4458-be8d-8d6922001217" (UID: "8e76d3f0-c1f4-4458-be8d-8d6922001217"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.156934 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.156915    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-data" pod "8e76d3f0-c1f4-4458-be8d-8d6922001217" (UID: "8e76d3f0-c1f4-4458-be8d-8d6922001217"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.165104    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-kube-api-access-sd7w7" pod "8e76d3f0-c1f4-4458-be8d-8d6922001217" (UID: "8e76d3f0-c1f4-4458-be8d-8d6922001217"). InnerVolumeSpecName "kube-api-access-sd7w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.165748    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-token" pod "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" (UID: "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.172444    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178121    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178152    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178167    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178181    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178195    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178209    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxq6d\" (UniqueName: \"kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-kube-api-access-dxq6d\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178225    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178240    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178253    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178266    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178279    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdv8q\" (UniqueName: \"kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-kube-api-access-rdv8q\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178295    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178309    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd7w7\" (UniqueName: \"kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-kube-api-access-sd7w7\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178324    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178340    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.178370    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45b4a4c-94e4-4897-8523-0e446e994bea-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.189963    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.189992    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.190009    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.190023    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e76d3f0-c1f4-4458-be8d-8d6922001217-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.190037    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.190049    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e76d3f0-c1f4-4458-be8d-8d6922001217-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.190064    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.190076    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.205466 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.190090    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45b4a4c-94e4-4897-8523-0e446e994bea-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.247626 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.246278    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:54.399423 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.365113    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"
Oct 30 13:05:54.469505 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.451462    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:54.469505 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.451792    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:05:54.486441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.485426    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-podinfo" pod "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" (UID: "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:05:54.584478 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.571527    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-envoy" pod "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" (UID: "bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:54.584478 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.572728    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 13:05:54.584478 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.574429    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:54.620518 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.620486    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"
Oct 30 13:05:54.666557 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:05:54.624770    4522 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.596s"
Oct 30 13:05:54.670391 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.666947    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 13:05:54.670391 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.666978    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z" event={"ID":"b45b4a4c-94e4-4897-8523-0e446e994bea","Type":"ContainerDied","Data":"fbbfb00b2d89d99029f20d1bf673c3ee1a824e4089b1d83759b118587b67519e"}
Oct 30 13:05:54.682465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.678590    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss" event={"ID":"4e6f2bb2-7f4c-4672-a201-9b822bea3687","Type":"ContainerDied","Data":"17a02fe97965584ab5e181e14177604d3e3e0e0dc6975e2092a8691a1f23f3c8"}
Oct 30 13:05:54.694481 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.694428    4522 scope.go:122] "RemoveContainer" containerID="72626aae9e9e3a957ae9125660854ec173922e4c29576a7cf007e08f2f7f87ff"
Oct 30 13:05:54.708769 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.708732    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.735387 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.709169    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:05:54.841104 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.838028    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:55.000437 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.921461    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:55.000437 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:54.921542    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:55.059400 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.059334    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:05:55.108943 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.073972    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:55.213500 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.112863    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:55.242529 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.142740    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:55.272465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.143860    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:55.385376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.144464    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:55.385376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.189329    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:55.385376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.231620    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:55.385376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.244947    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"
Oct 30 13:05:55.385376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.289929    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"
Oct 30 13:05:55.393433 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.391610    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz" event={"ID":"8e76d3f0-c1f4-4458-be8d-8d6922001217","Type":"ContainerDied","Data":"f03bce4d9878e9772b485387878b85e9eed8664eaf832672f1b10ddaee662773"}
Oct 30 13:05:55.393433 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.391698    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj" event={"ID":"bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef","Type":"ContainerDied","Data":"90c05e637c4a87dcbe36afb5c4b84e248cbd0003bff10d251a4fce34b86db41a"}
Oct 30 13:05:55.522377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.454368    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:55.522377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.477432    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:05:55.548715 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.537420    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:55.670377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.655803    4522 scope.go:122] "RemoveContainer" containerID="4e45cdbd858a3ff5675c2dae4a6c12621fc1d1af7b42f4700eb8aee7b45a9f00"
Oct 30 13:05:55.673175 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.673129    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:55.786577 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:55.786520    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded"
Oct 30 13:05:56.079656 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:56.074659    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:56.079656 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:56.075972    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:56.261964 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:56.261926    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"]
Oct 30 13:05:56.355700 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:05:56.355605    4522 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45b4a4c_94e4_4897_8523_0e446e994bea.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e76d3f0_c1f4_4458_be8d_8d6922001217.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6f2bb2_7f4c_4672_a201_9b822bea3687.slice/crio-17a02fe97965584ab5e181e14177604d3e3e0e0dc6975e2092a8691a1f23f3c8\": RecentStats: unable to find data in memory cache]"
Oct 30 13:05:56.715641 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:56.325160    4522 scope.go:122] "RemoveContainer" containerID="2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f"
Oct 30 13:05:56.739387 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:56.325465    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:56.759562 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:56.325506    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 13:05:56.771409 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:56.325541    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 13:05:56.771763 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:56.324106    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:05:56.772002 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:56.563884    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 13:05:56.856296 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:56.856250    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:57.075147 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.075105    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 13:05:57.080972 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.080936    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:57.085961 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.085928    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:57.106114 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.106063    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:57.109240 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.109217    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-rk86z"]
Oct 30 13:05:57.297377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.275593    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"]
Oct 30 13:05:57.387432 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.360153    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:05:57.494836 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.494802    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-r4tss"]
Oct 30 13:05:57.548626 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.548592    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"]
Oct 30 13:05:57.548900 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.548883    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-m9drz"]
Oct 30 13:05:57.589120 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.589089    4522 scope.go:122] "RemoveContainer" containerID="47f88fd4d522d009d64c4a880d593e7578975e00ed156b3e934a294a32a4a3a9"
Oct 30 13:05:57.605430 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.589993    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:05:57.661491 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.661410    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"]
Oct 30 13:05:57.713203 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.706744    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj"]
Oct 30 13:05:57.717553 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.717530    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:05:57.718072 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.718053    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:05:57.718215 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.709594    4522 scope.go:122] "RemoveContainer" containerID="47f88fd4d522d009d64c4a880d593e7578975e00ed156b3e934a294a32a4a3a9"
Oct 30 13:05:57.718394 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.718256    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:05:57.718544 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.718530    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq"
Oct 30 13:05:57.718686 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.718674    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:05:57.732468 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.729715    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:05:57.733498 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:05:57.733443    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f\": container with ID starting with 2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f not found: ID does not exist" containerID="2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f"
Oct 30 13:05:57.733498 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:05:57.733478    4522 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f\": container with ID starting with 2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f not found: ID does not exist"
Oct 30 13:05:57.733867 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.733839    4522 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f\": container with ID starting with 2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f not found: ID does not exist"
Oct 30 13:05:57.745926 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:05:57.745884    4522 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj_openshift-ingress_bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef_0 in pod sandbox 90c05e637c4a87dcbe36afb5c4b84e248cbd0003bff10d251a4fce34b86db41a from index: no such id: '47f88fd4d522d009d64c4a880d593e7578975e00ed156b3e934a294a32a4a3a9'" containerID="47f88fd4d522d009d64c4a880d593e7578975e00ed156b3e934a294a32a4a3a9"
Oct 30 13:05:57.746095 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:05:57.746016    4522 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w2-105228-openshift-default-7cfc8cdfb5-qzqlj_openshift-ingress_bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef_0 in pod sandbox 90c05e637c4a87dcbe36afb5c4b84e248cbd0003bff10d251a4fce34b86db41a from index: no such id: '47f88fd4d522d009d64c4a880d593e7578975e00ed156b3e934a294a32a4a3a9'" containerID="47f88fd4d522d009d64c4a880d593e7578975e00ed156b3e934a294a32a4a3a9"
Oct 30 13:05:57.746095 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.746051    4522 scope.go:122] "RemoveContainer" containerID="2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f"
Oct 30 13:05:57.746496 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:05:57.746467    4522 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f\": rpc error: code = NotFound desc = could not find container \"2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f\": container with ID starting with 2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f not found: ID does not exist" containerID="2896269a1f0cacdbf1bb6f153c80c42fffebe80abe31ad246eb47beb30db3d7f"
Oct 30 13:05:57.751574 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:57.751551    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:05:58.334582 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:58.334541    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" path="/var/lib/kubelet/pods/4e6f2bb2-7f4c-4672-a201-9b822bea3687/volumes"
Oct 30 13:05:58.335092 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:58.335077    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" path="/var/lib/kubelet/pods/8e76d3f0-c1f4-4458-be8d-8d6922001217/volumes"
Oct 30 13:05:58.335532 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:58.335519    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" path="/var/lib/kubelet/pods/b45b4a4c-94e4-4897-8523-0e446e994bea/volumes"
Oct 30 13:05:58.336243 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:58.336228    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" path="/var/lib/kubelet/pods/bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef/volumes"
Oct 30 13:05:58.613899 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:05:58.613823    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:00.408389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:00.405677    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:02.303202 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:02.303165    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:03.522969 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:03.522936    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:04.678524 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:04.678468    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:05.153868 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:05.153833    4522 generic.go:356] "Generic (PLEG): container finished" podID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerID="78065e469c98a58f34a7f18f8c4890205c0af25924b3ae184c8b93a90631063a" exitCode=0
Oct 30 13:06:05.154028 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:05.153899    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" event={"ID":"cb69b573-fa0e-4789-80e7-8df8caef1cb7","Type":"ContainerDied","Data":"78065e469c98a58f34a7f18f8c4890205c0af25924b3ae184c8b93a90631063a"}
Oct 30 13:06:05.154272 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:05.154259    4522 scope.go:122] "RemoveContainer" containerID="78065e469c98a58f34a7f18f8c4890205c0af25924b3ae184c8b93a90631063a"
Oct 30 13:06:05.155890 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:05.155856    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:06:05.155996 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:05.155941    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:06:05.155996 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:05.155976    4522 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:06:05.336490 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:05.336404    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:05.464479 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:05.464338    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:06.161983 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:06.161944    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" event={"ID":"cb69b573-fa0e-4789-80e7-8df8caef1cb7","Type":"ContainerStarted","Data":"b7baac47ed14a6de54425781dab6471a975832ad3f31d1696147905df164e6f8"}
Oct 30 13:06:06.843833 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:06.843800    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:06.935222 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:06.935190    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:06:06.935461 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:06.935433    4522 patch_prober.go:29] interesting pod/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 13:06:06.935555 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:06.935492    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 13:06:07.546861 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:07.544134    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:07.631808 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:07.631763    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:07.731545 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:07.731505    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:07.936535 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:07.936461    4522 patch_prober.go:29] interesting pod/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 13:06:07.936535 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:07.936508    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 13:06:08.156472 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:08.156440    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:08.935382 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:08.935323    4522 patch_prober.go:29] interesting pod/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 13:06:08.935765 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:08.935403    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 13:06:09.925876 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:09.925845    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:09.936062 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:09.936037    4522 patch_prober.go:29] interesting pod/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 13:06:09.936436 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:09.936090    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 13:06:10.593274 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:10.593236    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:10.936158 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:10.936086    4522 patch_prober.go:29] interesting pod/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 13:06:10.936158 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:10.936137    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 13:06:11.742083 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:11.742050    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:06:11.818329 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:11.818297    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:11.936510 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:11.936470    4522 patch_prober.go:29] interesting pod/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 13:06:11.937108 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:11.937076    4522 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 13:06:12.939950 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:12.939924    4522 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:06:12.941168 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:12.941143    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:06:13.693794 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:13.693765    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:14.146226 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:14.146188    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:14.276347 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:14.276312    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:14.600898 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:14.600870    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"]
Oct 30 13:06:14.601082 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:14.601049    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv" podUID="f382c2bb-0885-48f3-9ac1-09585a2b1c36" containerName="istio-proxy" containerID="cri-o://576ca5cc5e9688700d553d47b2e7923296909152b625b7f0033cb576ca0f982d" gracePeriod=30
Oct 30 13:06:14.728239 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:14.728197    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:15.539587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:15.539551    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"]
Oct 30 13:06:15.550996 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:15.550963    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"]
Oct 30 13:06:15.555980 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:15.555951    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"]
Oct 30 13:06:15.564599 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:15.564576    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"]
Oct 30 13:06:15.659992 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:15.659968    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:15.771941 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:15.771907    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:17.697486 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:17.697440    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:19.522101 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.522058    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:19.872642 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.872617    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:06:19.955240 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.955205    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-credential-socket\") pod \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") "
Oct 30 13:06:19.955413 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.955266    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-token\" (UniqueName: \"kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-token\") pod \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") "
Oct 30 13:06:19.955413 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.955365    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-data\") pod \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") "
Oct 30 13:06:19.955556 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.955420    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istiod-ca-cert\") pod \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") "
Oct 30 13:06:19.955556 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.955523    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-envoy\") pod \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") "
Oct 30 13:06:19.955556 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.955527    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-credential-socket" pod "f382c2bb-0885-48f3-9ac1-09585a2b1c36" (UID: "f382c2bb-0885-48f3-9ac1-09585a2b1c36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:19.955723 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.955564    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-socket\") pod \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") "
Oct 30 13:06:19.955723 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.955627    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-podinfo\") pod \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") "
Oct 30 13:06:19.955842 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.955735    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-certs\") pod \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") "
Oct 30 13:06:19.955842 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.955764    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-kube-api-access-csmg6\" (UniqueName: \"kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-kube-api-access-csmg6\") pod \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\" (UID: \"f382c2bb-0885-48f3-9ac1-09585a2b1c36\") "
Oct 30 13:06:19.955949 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.955861    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istiod-ca-cert" pod "f382c2bb-0885-48f3-9ac1-09585a2b1c36" (UID: "f382c2bb-0885-48f3-9ac1-09585a2b1c36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:19.956007 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.955948    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-data" pod "f382c2bb-0885-48f3-9ac1-09585a2b1c36" (UID: "f382c2bb-0885-48f3-9ac1-09585a2b1c36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:19.956311 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.956286    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-socket" pod "f382c2bb-0885-48f3-9ac1-09585a2b1c36" (UID: "f382c2bb-0885-48f3-9ac1-09585a2b1c36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:19.956576 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.956542    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:19.956576 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.956565    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:19.956724 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.956579    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:19.956849 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.956829    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-certs" pod "f382c2bb-0885-48f3-9ac1-09585a2b1c36" (UID: "f382c2bb-0885-48f3-9ac1-09585a2b1c36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:19.958784 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.958757    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-podinfo" pod "f382c2bb-0885-48f3-9ac1-09585a2b1c36" (UID: "f382c2bb-0885-48f3-9ac1-09585a2b1c36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:06:19.958884 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.958767    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-token" pod "f382c2bb-0885-48f3-9ac1-09585a2b1c36" (UID: "f382c2bb-0885-48f3-9ac1-09585a2b1c36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:19.958884 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.958784    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-envoy" pod "f382c2bb-0885-48f3-9ac1-09585a2b1c36" (UID: "f382c2bb-0885-48f3-9ac1-09585a2b1c36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:19.958884 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:19.958806    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-kube-api-access-csmg6" pod "f382c2bb-0885-48f3-9ac1-09585a2b1c36" (UID: "f382c2bb-0885-48f3-9ac1-09585a2b1c36"). InnerVolumeSpecName "kube-api-access-csmg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:20.057782 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.057721    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.057782 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.057745    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csmg6\" (UniqueName: \"kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-kube-api-access-csmg6\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.057782 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.057755    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.057782 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.057763    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.057782 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.057770    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382c2bb-0885-48f3-9ac1-09585a2b1c36-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.057782 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.057778    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f382c2bb-0885-48f3-9ac1-09585a2b1c36-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:20.368332 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.368300    4522 generic.go:356] "Generic (PLEG): container finished" podID="f382c2bb-0885-48f3-9ac1-09585a2b1c36" containerID="576ca5cc5e9688700d553d47b2e7923296909152b625b7f0033cb576ca0f982d" exitCode=0
Oct 30 13:06:20.368455 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.368344    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"
Oct 30 13:06:20.368498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.368381    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv" event={"ID":"f382c2bb-0885-48f3-9ac1-09585a2b1c36","Type":"ContainerDied","Data":"576ca5cc5e9688700d553d47b2e7923296909152b625b7f0033cb576ca0f982d"}
Oct 30 13:06:20.368542 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.368518    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv" event={"ID":"f382c2bb-0885-48f3-9ac1-09585a2b1c36","Type":"ContainerDied","Data":"0e27f72a63e9bdce99f4d065ce3d34fca861e53d9d8bc41cbf1b20969b158d0a"}
Oct 30 13:06:20.368576 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.368549    4522 scope.go:122] "RemoveContainer" containerID="576ca5cc5e9688700d553d47b2e7923296909152b625b7f0033cb576ca0f982d"
Oct 30 13:06:20.378105 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.378085    4522 scope.go:122] "RemoveContainer" containerID="576ca5cc5e9688700d553d47b2e7923296909152b625b7f0033cb576ca0f982d"
Oct 30 13:06:20.378394 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:06:20.378374    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576ca5cc5e9688700d553d47b2e7923296909152b625b7f0033cb576ca0f982d\": container with ID starting with 576ca5cc5e9688700d553d47b2e7923296909152b625b7f0033cb576ca0f982d not found: ID does not exist" containerID="576ca5cc5e9688700d553d47b2e7923296909152b625b7f0033cb576ca0f982d"
Oct 30 13:06:20.378454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.378399    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576ca5cc5e9688700d553d47b2e7923296909152b625b7f0033cb576ca0f982d"} err="failed to get container status \"576ca5cc5e9688700d553d47b2e7923296909152b625b7f0033cb576ca0f982d\": rpc error: code = NotFound desc = could not find container \"576ca5cc5e9688700d553d47b2e7923296909152b625b7f0033cb576ca0f982d\": container with ID starting with 576ca5cc5e9688700d553d47b2e7923296909152b625b7f0033cb576ca0f982d not found: ID does not exist"
Oct 30 13:06:20.437921 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.437900    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"]
Oct 30 13:06:20.440608 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:20.440589    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-f73c9f-openshift-default-8476dd64cc-7hkrv"]
Oct 30 13:06:21.136697 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:06:21.136659    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:06:21.414579 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:21.414506    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:21.533617 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:21.533579    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:21.664478 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:21.663310    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:22.336492 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:22.336458    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f382c2bb-0885-48f3-9ac1-09585a2b1c36" path="/var/lib/kubelet/pods/f382c2bb-0885-48f3-9ac1-09585a2b1c36/volumes"
Oct 30 13:06:23.255447 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:23.255414    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:25.560925 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:25.560893    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:25.670819 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:25.670781    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:25.805772 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:25.805729    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:27.831993 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:27.831956    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:29.749771 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:29.749736    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:30.563730 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.563694    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main" containerID="cri-o://f4c954dc8c41bae46b3c16842bc1cf8917474a372a0c1e2eba872295b96bc7bc" gracePeriod=45
Oct 30 13:06:30.578751 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.578713    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main" containerID="cri-o://3e69ea35646f68e0bf199206f436fa45d5fb9cffb31de729d3baa7a1fcd9d583" gracePeriod=45
Oct 30 13:06:30.578751 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.578715    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main" containerID="cri-o://5c1c9d2eaed8f08aebd0f250cdee900ce150c7ce91ea102e4bdfd84f85663a2a" gracePeriod=45
Oct 30 13:06:30.584028 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.583997    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main" containerID="cri-o://2a389ea1e14df53b9870ac7fad2b3bea73c8076fe00bec3028236a5f8dbc6f19" gracePeriod=45
Oct 30 13:06:30.782925 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.782889    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:30.876598 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.876568    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:06:30.891484 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.891454    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:06:30.908782 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.908760    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:06:30.912522 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.912503    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:06:30.998805 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.998767    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-dshm\") pod \"dc1616ab-319c-4608-893b-fdcc65631488\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") "
Oct 30 13:06:30.998969 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.998823    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-dshm\") pod \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") "
Oct 30 13:06:30.998969 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.998857    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-tmp-dir\") pod \"f59f4327-f9c2-4445-97b6-63a99043d625\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") "
Oct 30 13:06:30.998969 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.998894    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-model-cache\") pod \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") "
Oct 30 13:06:30.998969 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.998936    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-home\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-home\") pod \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") "
Oct 30 13:06:30.999223 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.998995    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc1616ab-319c-4608-893b-fdcc65631488-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1616ab-319c-4608-893b-fdcc65631488-tls-certs\") pod \"dc1616ab-319c-4608-893b-fdcc65631488\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") "
Oct 30 13:06:30.999223 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.999045    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-model-cache\") pod \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") "
Oct 30 13:06:30.999223 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.999079    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2423ab7f-0469-463d-9aaf-b17895f4c4a6-kube-api-access-6vsnv\" (UniqueName: \"kubernetes.io/projected/2423ab7f-0469-463d-9aaf-b17895f4c4a6-kube-api-access-6vsnv\") pod \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") "
Oct 30 13:06:30.999223 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.999149    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tls-certs\" (UniqueName: \"kubernetes.io/secret/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tls-certs\") pod \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") "
Oct 30 13:06:30.999223 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.999166    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-tmp-dir" pod "f59f4327-f9c2-4445-97b6-63a99043d625" (UID: "f59f4327-f9c2-4445-97b6-63a99043d625"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:30.999223 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.999202    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-home\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-home\") pod \"dc1616ab-319c-4608-893b-fdcc65631488\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") "
Oct 30 13:06:30.999580 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.999228    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tmp-dir\") pod \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") "
Oct 30 13:06:30.999580 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.999257    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-model-cache" pod "849bf94f-d57d-4f16-8494-09bbbf8e3db9" (UID: "849bf94f-d57d-4f16-8494-09bbbf8e3db9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:30.999580 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.999276    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1616ab-319c-4608-893b-fdcc65631488-kube-api-access-qhbpr\" (UniqueName: \"kubernetes.io/projected/dc1616ab-319c-4608-893b-fdcc65631488-kube-api-access-qhbpr\") pod \"dc1616ab-319c-4608-893b-fdcc65631488\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") "
Oct 30 13:06:30.999580 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.999344    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-model-cache" pod "2423ab7f-0469-463d-9aaf-b17895f4c4a6" (UID: "2423ab7f-0469-463d-9aaf-b17895f4c4a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:30.999580 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.999411    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-home" pod "849bf94f-d57d-4f16-8494-09bbbf8e3db9" (UID: "849bf94f-d57d-4f16-8494-09bbbf8e3db9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:30.999580 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.999423    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tmp-dir\") pod \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") "
Oct 30 13:06:30.999899 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.999662    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tmp-dir" pod "849bf94f-d57d-4f16-8494-09bbbf8e3db9" (UID: "849bf94f-d57d-4f16-8494-09bbbf8e3db9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:30.999899 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:30.999844    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-home" pod "dc1616ab-319c-4608-893b-fdcc65631488" (UID: "dc1616ab-319c-4608-893b-fdcc65631488"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:31.000042 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000022    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tmp-dir" pod "2423ab7f-0469-463d-9aaf-b17895f4c4a6" (UID: "2423ab7f-0469-463d-9aaf-b17895f4c4a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:31.000111 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000066    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-tmp-dir\") pod \"dc1616ab-319c-4608-893b-fdcc65631488\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") "
Oct 30 13:06:31.000178 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000113    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-dshm\") pod \"f59f4327-f9c2-4445-97b6-63a99043d625\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") "
Oct 30 13:06:31.000178 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000167    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f59f4327-f9c2-4445-97b6-63a99043d625-tls-certs\" (UniqueName: \"kubernetes.io/secret/f59f4327-f9c2-4445-97b6-63a99043d625-tls-certs\") pod \"f59f4327-f9c2-4445-97b6-63a99043d625\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") "
Oct 30 13:06:31.000294 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000199    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849bf94f-d57d-4f16-8494-09bbbf8e3db9-kube-api-access-npjd2\" (UniqueName: \"kubernetes.io/projected/849bf94f-d57d-4f16-8494-09bbbf8e3db9-kube-api-access-npjd2\") pod \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") "
Oct 30 13:06:31.000294 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000237    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59f4327-f9c2-4445-97b6-63a99043d625-kube-api-access-lchfr\" (UniqueName: \"kubernetes.io/projected/f59f4327-f9c2-4445-97b6-63a99043d625-kube-api-access-lchfr\") pod \"f59f4327-f9c2-4445-97b6-63a99043d625\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") "
Oct 30 13:06:31.000453 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000327    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tls-certs\") pod \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") "
Oct 30 13:06:31.000453 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000343    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-tmp-dir" pod "dc1616ab-319c-4608-893b-fdcc65631488" (UID: "dc1616ab-319c-4608-893b-fdcc65631488"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:31.000453 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000374    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-home\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-home\") pod \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\" (UID: \"2423ab7f-0469-463d-9aaf-b17895f4c4a6\") "
Oct 30 13:06:31.000643 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000461    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-model-cache\") pod \"dc1616ab-319c-4608-893b-fdcc65631488\" (UID: \"dc1616ab-319c-4608-893b-fdcc65631488\") "
Oct 30 13:06:31.000643 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000554    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-dshm\") pod \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\" (UID: \"849bf94f-d57d-4f16-8494-09bbbf8e3db9\") "
Oct 30 13:06:31.000801 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000643    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-home\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-home\") pod \"f59f4327-f9c2-4445-97b6-63a99043d625\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") "
Oct 30 13:06:31.000801 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000679    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-model-cache\") pod \"f59f4327-f9c2-4445-97b6-63a99043d625\" (UID: \"f59f4327-f9c2-4445-97b6-63a99043d625\") "
Oct 30 13:06:31.001465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.001344    4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-tmp-dir\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.001465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.001387    4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-model-cache\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.001465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.001399    4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-home\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.001465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.001411    4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-model-cache\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.001465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.001421    4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-home\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.001465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.001433    4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tmp-dir\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.001891 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.001571    4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tmp-dir\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.001891 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.001587    4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-tmp-dir\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.001998 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.000557    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-home" pod "2423ab7f-0469-463d-9aaf-b17895f4c4a6" (UID: "2423ab7f-0469-463d-9aaf-b17895f4c4a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:31.001998 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.001815    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-model-cache" pod "f59f4327-f9c2-4445-97b6-63a99043d625" (UID: "f59f4327-f9c2-4445-97b6-63a99043d625"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:31.001998 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.001971    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tls-certs" pod "849bf94f-d57d-4f16-8494-09bbbf8e3db9" (UID: "849bf94f-d57d-4f16-8494-09bbbf8e3db9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:31.002230 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.002207    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-home" pod "f59f4327-f9c2-4445-97b6-63a99043d625" (UID: "f59f4327-f9c2-4445-97b6-63a99043d625"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:31.002297 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.002273    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-dshm" pod "dc1616ab-319c-4608-893b-fdcc65631488" (UID: "dc1616ab-319c-4608-893b-fdcc65631488"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:31.002370 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.002301    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-model-cache" pod "dc1616ab-319c-4608-893b-fdcc65631488" (UID: "dc1616ab-319c-4608-893b-fdcc65631488"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:31.002445 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.002386    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1616ab-319c-4608-893b-fdcc65631488-kube-api-access-qhbpr" pod "dc1616ab-319c-4608-893b-fdcc65631488" (UID: "dc1616ab-319c-4608-893b-fdcc65631488"). InnerVolumeSpecName "kube-api-access-qhbpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:31.003283 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.003230    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2423ab7f-0469-463d-9aaf-b17895f4c4a6-kube-api-access-6vsnv" pod "2423ab7f-0469-463d-9aaf-b17895f4c4a6" (UID: "2423ab7f-0469-463d-9aaf-b17895f4c4a6"). InnerVolumeSpecName "kube-api-access-6vsnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:31.003738 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.003709    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1616ab-319c-4608-893b-fdcc65631488-tls-certs" pod "dc1616ab-319c-4608-893b-fdcc65631488" (UID: "dc1616ab-319c-4608-893b-fdcc65631488"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:31.003976 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.003949    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-dshm" pod "2423ab7f-0469-463d-9aaf-b17895f4c4a6" (UID: "2423ab7f-0469-463d-9aaf-b17895f4c4a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:31.004447 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.004421    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-dshm" pod "f59f4327-f9c2-4445-97b6-63a99043d625" (UID: "f59f4327-f9c2-4445-97b6-63a99043d625"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:31.004560 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.004453    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849bf94f-d57d-4f16-8494-09bbbf8e3db9-kube-api-access-npjd2" pod "849bf94f-d57d-4f16-8494-09bbbf8e3db9" (UID: "849bf94f-d57d-4f16-8494-09bbbf8e3db9"). InnerVolumeSpecName "kube-api-access-npjd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:31.004660 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.004636    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59f4327-f9c2-4445-97b6-63a99043d625-kube-api-access-lchfr" pod "f59f4327-f9c2-4445-97b6-63a99043d625" (UID: "f59f4327-f9c2-4445-97b6-63a99043d625"). InnerVolumeSpecName "kube-api-access-lchfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:31.005002 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.004978    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-dshm" pod "849bf94f-d57d-4f16-8494-09bbbf8e3db9" (UID: "849bf94f-d57d-4f16-8494-09bbbf8e3db9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:31.005631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.005608    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f59f4327-f9c2-4445-97b6-63a99043d625-tls-certs" pod "f59f4327-f9c2-4445-97b6-63a99043d625" (UID: "f59f4327-f9c2-4445-97b6-63a99043d625"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:31.005713 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.005676    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tls-certs" pod "2423ab7f-0469-463d-9aaf-b17895f4c4a6" (UID: "2423ab7f-0469-463d-9aaf-b17895f4c4a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:31.039058 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.039026    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:31.103398 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103306    4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-model-cache\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103398 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103342    4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849bf94f-d57d-4f16-8494-09bbbf8e3db9-dshm\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103398 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103369    4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-home\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103398 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103382    4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-model-cache\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103398 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103396    4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1616ab-319c-4608-893b-fdcc65631488-dshm\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103408    4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-dshm\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103421    4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1616ab-319c-4608-893b-fdcc65631488-tls-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103435    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vsnv\" (UniqueName: \"kubernetes.io/projected/2423ab7f-0469-463d-9aaf-b17895f4c4a6-kube-api-access-6vsnv\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103448    4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849bf94f-d57d-4f16-8494-09bbbf8e3db9-tls-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103461    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhbpr\" (UniqueName: \"kubernetes.io/projected/dc1616ab-319c-4608-893b-fdcc65631488-kube-api-access-qhbpr\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103473    4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f59f4327-f9c2-4445-97b6-63a99043d625-dshm\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103486    4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f59f4327-f9c2-4445-97b6-63a99043d625-tls-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103499    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npjd2\" (UniqueName: \"kubernetes.io/projected/849bf94f-d57d-4f16-8494-09bbbf8e3db9-kube-api-access-npjd2\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103513    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lchfr\" (UniqueName: \"kubernetes.io/projected/f59f4327-f9c2-4445-97b6-63a99043d625-kube-api-access-lchfr\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103526    4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2423ab7f-0469-463d-9aaf-b17895f4c4a6-tls-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.103780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.103538    4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2423ab7f-0469-463d-9aaf-b17895f4c4a6-home\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.444304 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.444219    4522 generic.go:356] "Generic (PLEG): container finished" podID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerID="3e69ea35646f68e0bf199206f436fa45d5fb9cffb31de729d3baa7a1fcd9d583" exitCode=0
Oct 30 13:06:31.444304 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.444272    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"
Oct 30 13:06:31.444581 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.444315    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" event={"ID":"849bf94f-d57d-4f16-8494-09bbbf8e3db9","Type":"ContainerDied","Data":"3e69ea35646f68e0bf199206f436fa45d5fb9cffb31de729d3baa7a1fcd9d583"}
Oct 30 13:06:31.444581 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.444379    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb" event={"ID":"849bf94f-d57d-4f16-8494-09bbbf8e3db9","Type":"ContainerDied","Data":"adf8872d0beeff685b6774f06db745f81e7b7ed53a936eab32419fcefa38b53c"}
Oct 30 13:06:31.444581 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.444402    4522 scope.go:122] "RemoveContainer" containerID="3e69ea35646f68e0bf199206f436fa45d5fb9cffb31de729d3baa7a1fcd9d583"
Oct 30 13:06:31.445858 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.445823    4522 generic.go:356] "Generic (PLEG): container finished" podID="dc1616ab-319c-4608-893b-fdcc65631488" containerID="f4c954dc8c41bae46b3c16842bc1cf8917474a372a0c1e2eba872295b96bc7bc" exitCode=0
Oct 30 13:06:31.445963 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.445889    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"
Oct 30 13:06:31.445963 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.445916    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" event={"ID":"dc1616ab-319c-4608-893b-fdcc65631488","Type":"ContainerDied","Data":"f4c954dc8c41bae46b3c16842bc1cf8917474a372a0c1e2eba872295b96bc7bc"}
Oct 30 13:06:31.445963 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.445954    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5" event={"ID":"dc1616ab-319c-4608-893b-fdcc65631488","Type":"ContainerDied","Data":"123a0c36adf0bb5cfeb4d7cc67d60a3579e2f9cdff55bf8423c96e28e1a6e3ce"}
Oct 30 13:06:31.450652 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.450625    4522 generic.go:356] "Generic (PLEG): container finished" podID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerID="2a389ea1e14df53b9870ac7fad2b3bea73c8076fe00bec3028236a5f8dbc6f19" exitCode=0
Oct 30 13:06:31.450789 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.450685    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"
Oct 30 13:06:31.450789 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.450706    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" event={"ID":"2423ab7f-0469-463d-9aaf-b17895f4c4a6","Type":"ContainerDied","Data":"2a389ea1e14df53b9870ac7fad2b3bea73c8076fe00bec3028236a5f8dbc6f19"}
Oct 30 13:06:31.450789 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.450742    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r" event={"ID":"2423ab7f-0469-463d-9aaf-b17895f4c4a6","Type":"ContainerDied","Data":"2a3a9b18681f11fa4c08e486238492ab181ed7f73e4d98604293477c18b20939"}
Oct 30 13:06:31.452196 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.452173    4522 generic.go:356] "Generic (PLEG): container finished" podID="f59f4327-f9c2-4445-97b6-63a99043d625" containerID="5c1c9d2eaed8f08aebd0f250cdee900ce150c7ce91ea102e4bdfd84f85663a2a" exitCode=0
Oct 30 13:06:31.452305 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.452221    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"
Oct 30 13:06:31.452305 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.452245    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" event={"ID":"f59f4327-f9c2-4445-97b6-63a99043d625","Type":"ContainerDied","Data":"5c1c9d2eaed8f08aebd0f250cdee900ce150c7ce91ea102e4bdfd84f85663a2a"}
Oct 30 13:06:31.452305 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.452274    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt" event={"ID":"f59f4327-f9c2-4445-97b6-63a99043d625","Type":"ContainerDied","Data":"b56ae8666115a827844578dc6550dc021a8eed8c9e5b8fbd841e0fd8ea29b3bc"}
Oct 30 13:06:31.455919 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.455902    4522 scope.go:122] "RemoveContainer" containerID="3e69ea35646f68e0bf199206f436fa45d5fb9cffb31de729d3baa7a1fcd9d583"
Oct 30 13:06:31.456294 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:06:31.456269    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e69ea35646f68e0bf199206f436fa45d5fb9cffb31de729d3baa7a1fcd9d583\": container with ID starting with 3e69ea35646f68e0bf199206f436fa45d5fb9cffb31de729d3baa7a1fcd9d583 not found: ID does not exist" containerID="3e69ea35646f68e0bf199206f436fa45d5fb9cffb31de729d3baa7a1fcd9d583"
Oct 30 13:06:31.456423 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.456301    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e69ea35646f68e0bf199206f436fa45d5fb9cffb31de729d3baa7a1fcd9d583"} err="failed to get container status \"3e69ea35646f68e0bf199206f436fa45d5fb9cffb31de729d3baa7a1fcd9d583\": rpc error: code = NotFound desc = could not find container \"3e69ea35646f68e0bf199206f436fa45d5fb9cffb31de729d3baa7a1fcd9d583\": container with ID starting with 3e69ea35646f68e0bf199206f436fa45d5fb9cffb31de729d3baa7a1fcd9d583 not found: ID does not exist"
Oct 30 13:06:31.456423 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.456322    4522 scope.go:122] "RemoveContainer" containerID="f4c954dc8c41bae46b3c16842bc1cf8917474a372a0c1e2eba872295b96bc7bc"
Oct 30 13:06:31.466821 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.466802    4522 scope.go:122] "RemoveContainer" containerID="f4c954dc8c41bae46b3c16842bc1cf8917474a372a0c1e2eba872295b96bc7bc"
Oct 30 13:06:31.467951 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:06:31.467924    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c954dc8c41bae46b3c16842bc1cf8917474a372a0c1e2eba872295b96bc7bc\": container with ID starting with f4c954dc8c41bae46b3c16842bc1cf8917474a372a0c1e2eba872295b96bc7bc not found: ID does not exist" containerID="f4c954dc8c41bae46b3c16842bc1cf8917474a372a0c1e2eba872295b96bc7bc"
Oct 30 13:06:31.468041 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.467956    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c954dc8c41bae46b3c16842bc1cf8917474a372a0c1e2eba872295b96bc7bc"} err="failed to get container status \"f4c954dc8c41bae46b3c16842bc1cf8917474a372a0c1e2eba872295b96bc7bc\": rpc error: code = NotFound desc = could not find container \"f4c954dc8c41bae46b3c16842bc1cf8917474a372a0c1e2eba872295b96bc7bc\": container with ID starting with f4c954dc8c41bae46b3c16842bc1cf8917474a372a0c1e2eba872295b96bc7bc not found: ID does not exist"
Oct 30 13:06:31.468041 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.467973    4522 scope.go:122] "RemoveContainer" containerID="2a389ea1e14df53b9870ac7fad2b3bea73c8076fe00bec3028236a5f8dbc6f19"
Oct 30 13:06:31.475570 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.475555    4522 scope.go:122] "RemoveContainer" containerID="2a389ea1e14df53b9870ac7fad2b3bea73c8076fe00bec3028236a5f8dbc6f19"
Oct 30 13:06:31.475802 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:06:31.475782    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a389ea1e14df53b9870ac7fad2b3bea73c8076fe00bec3028236a5f8dbc6f19\": container with ID starting with 2a389ea1e14df53b9870ac7fad2b3bea73c8076fe00bec3028236a5f8dbc6f19 not found: ID does not exist" containerID="2a389ea1e14df53b9870ac7fad2b3bea73c8076fe00bec3028236a5f8dbc6f19"
Oct 30 13:06:31.475847 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.475808    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a389ea1e14df53b9870ac7fad2b3bea73c8076fe00bec3028236a5f8dbc6f19"} err="failed to get container status \"2a389ea1e14df53b9870ac7fad2b3bea73c8076fe00bec3028236a5f8dbc6f19\": rpc error: code = NotFound desc = could not find container \"2a389ea1e14df53b9870ac7fad2b3bea73c8076fe00bec3028236a5f8dbc6f19\": container with ID starting with 2a389ea1e14df53b9870ac7fad2b3bea73c8076fe00bec3028236a5f8dbc6f19 not found: ID does not exist"
Oct 30 13:06:31.475847 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.475821    4522 scope.go:122] "RemoveContainer" containerID="5c1c9d2eaed8f08aebd0f250cdee900ce150c7ce91ea102e4bdfd84f85663a2a"
Oct 30 13:06:31.483406 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.483346    4522 scope.go:122] "RemoveContainer" containerID="5c1c9d2eaed8f08aebd0f250cdee900ce150c7ce91ea102e4bdfd84f85663a2a"
Oct 30 13:06:31.483649 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:06:31.483621    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1c9d2eaed8f08aebd0f250cdee900ce150c7ce91ea102e4bdfd84f85663a2a\": container with ID starting with 5c1c9d2eaed8f08aebd0f250cdee900ce150c7ce91ea102e4bdfd84f85663a2a not found: ID does not exist" containerID="5c1c9d2eaed8f08aebd0f250cdee900ce150c7ce91ea102e4bdfd84f85663a2a"
Oct 30 13:06:31.483701 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.483654    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1c9d2eaed8f08aebd0f250cdee900ce150c7ce91ea102e4bdfd84f85663a2a"} err="failed to get container status \"5c1c9d2eaed8f08aebd0f250cdee900ce150c7ce91ea102e4bdfd84f85663a2a\": rpc error: code = NotFound desc = could not find container \"5c1c9d2eaed8f08aebd0f250cdee900ce150c7ce91ea102e4bdfd84f85663a2a\": container with ID starting with 5c1c9d2eaed8f08aebd0f250cdee900ce150c7ce91ea102e4bdfd84f85663a2a not found: ID does not exist"
Oct 30 13:06:31.495706 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.495687    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"]
Oct 30 13:06:31.507474 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.507453    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/distinct-2-w3-c81f42-kserve-75ff9f687f-mthg5"]
Oct 30 13:06:31.549414 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.549395    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"]
Oct 30 13:06:31.551967 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.551949    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/distinct-w3-c81f42-kserve-855cb9f656-nt4pb"]
Oct 30 13:06:31.603317 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.603292    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"]
Oct 30 13:06:31.607085 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.607065    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/premium-w3-c81f42-kserve-58f455bcdf-vvxvt"]
Oct 30 13:06:31.644071 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.644049    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"]
Oct 30 13:06:31.647771 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.647748    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c81f42/simulator-w3-c81f42-kserve-656c8f6ddd-v4b6r"]
Oct 30 13:06:31.692377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:31.691582    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:32.334069 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:32.334036    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" path="/var/lib/kubelet/pods/2423ab7f-0469-463d-9aaf-b17895f4c4a6/volumes"
Oct 30 13:06:32.334513 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:32.334497    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" path="/var/lib/kubelet/pods/849bf94f-d57d-4f16-8494-09bbbf8e3db9/volumes"
Oct 30 13:06:32.334905 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:32.334891    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1616ab-319c-4608-893b-fdcc65631488" path="/var/lib/kubelet/pods/dc1616ab-319c-4608-893b-fdcc65631488/volumes"
Oct 30 13:06:32.335488 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:32.335469    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" path="/var/lib/kubelet/pods/f59f4327-f9c2-4445-97b6-63a99043d625/volumes"
Oct 30 13:06:33.068364 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:33.068312    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:33.662781 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:33.662735    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:33.818591 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:33.818564    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:33.918599 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:33.918529    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:34.314579 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:34.314545    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:35.829833 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:35.829794    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:37.709109 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:37.709075    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:39.454493 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:39.454408    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:40.506712 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:40.506676    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:40.630642 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:40.630599    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:40.905960 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:40.905924    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:41.467476 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:41.467442    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:42.127100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:42.127050    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:44.621195 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:44.621164    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:46.343700 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:46.343660    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:46.484007 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:46.483858    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"]
Oct 30 13:06:46.576091 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:46.576058    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:48.119580 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:48.119550    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:49.823975 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:49.823942    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:49.925950 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:49.925910    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:51.133776 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:06:51.133737    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:06:51.523177 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:51.523148    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:52.556196 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:52.556158    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:52.706376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:52.704761    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:53.349786 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:53.349716    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:53.458651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:53.458616    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:53.768043 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:53.767955    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:54.700299 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:54.700268    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:55.714879 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:55.714841    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"]
Oct 30 13:06:55.715312 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:55.715058    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544" podUID="186f443b-4af4-40dd-ad37-71be0c18ebf4" containerName="maas-api" containerID="cri-o://2df051703d6bcdb36340d0e433b2ff22545fcb9342a1a4bcfa90e53c431c5720" gracePeriod=30
Oct 30 13:06:55.780981 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:55.780951    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:56.475878 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:56.475840    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:56.631380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:56.631322    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:56.755827 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:56.755753    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:57.039189 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.039164    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 13:06:57.060909 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.060885    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-maas-api-tls\") pod \"186f443b-4af4-40dd-ad37-71be0c18ebf4\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") "
Oct 30 13:06:57.061044 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.060974    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-metrics-certs\") pod \"186f443b-4af4-40dd-ad37-71be0c18ebf4\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") "
Oct 30 13:06:57.061044 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.061014    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186f443b-4af4-40dd-ad37-71be0c18ebf4-kube-api-access-vjv24\" (UniqueName: \"kubernetes.io/projected/186f443b-4af4-40dd-ad37-71be0c18ebf4-kube-api-access-vjv24\") pod \"186f443b-4af4-40dd-ad37-71be0c18ebf4\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") "
Oct 30 13:06:57.061172 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.061118    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186f443b-4af4-40dd-ad37-71be0c18ebf4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186f443b-4af4-40dd-ad37-71be0c18ebf4-openshift-service-ca\") pod \"186f443b-4af4-40dd-ad37-71be0c18ebf4\" (UID: \"186f443b-4af4-40dd-ad37-71be0c18ebf4\") "
Oct 30 13:06:57.061525 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.061495    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186f443b-4af4-40dd-ad37-71be0c18ebf4-openshift-service-ca" pod "186f443b-4af4-40dd-ad37-71be0c18ebf4" (UID: "186f443b-4af4-40dd-ad37-71be0c18ebf4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:57.061712 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.061691    4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186f443b-4af4-40dd-ad37-71be0c18ebf4-openshift-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:57.063165 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.063138    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-maas-api-tls" pod "186f443b-4af4-40dd-ad37-71be0c18ebf4" (UID: "186f443b-4af4-40dd-ad37-71be0c18ebf4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:57.063457 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.063422    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-metrics-certs" pod "186f443b-4af4-40dd-ad37-71be0c18ebf4" (UID: "186f443b-4af4-40dd-ad37-71be0c18ebf4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:57.063848 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.063827    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186f443b-4af4-40dd-ad37-71be0c18ebf4-kube-api-access-vjv24" pod "186f443b-4af4-40dd-ad37-71be0c18ebf4" (UID: "186f443b-4af4-40dd-ad37-71be0c18ebf4"). InnerVolumeSpecName "kube-api-access-vjv24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:57.146813 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.146780    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:57.162404 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.162380    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjv24\" (UniqueName: \"kubernetes.io/projected/186f443b-4af4-40dd-ad37-71be0c18ebf4-kube-api-access-vjv24\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:57.162404 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.162404    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-maas-api-tls\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:57.162534 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.162414    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186f443b-4af4-40dd-ad37-71be0c18ebf4-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:06:57.761562 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.761527    4522 generic.go:356] "Generic (PLEG): container finished" podID="186f443b-4af4-40dd-ad37-71be0c18ebf4" containerID="2df051703d6bcdb36340d0e433b2ff22545fcb9342a1a4bcfa90e53c431c5720" exitCode=0
Oct 30 13:06:57.761982 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.761570    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"
Oct 30 13:06:57.761982 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.761608    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544" event={"ID":"186f443b-4af4-40dd-ad37-71be0c18ebf4","Type":"ContainerDied","Data":"2df051703d6bcdb36340d0e433b2ff22545fcb9342a1a4bcfa90e53c431c5720"}
Oct 30 13:06:57.761982 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.761662    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544" event={"ID":"186f443b-4af4-40dd-ad37-71be0c18ebf4","Type":"ContainerDied","Data":"25fac7042e8940ec10d9caaa337a97887d64ca187d8d621f0a7eaceb1ec14615"}
Oct 30 13:06:57.761982 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.761692    4522 scope.go:122] "RemoveContainer" containerID="2df051703d6bcdb36340d0e433b2ff22545fcb9342a1a4bcfa90e53c431c5720"
Oct 30 13:06:57.772476 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.772454    4522 scope.go:122] "RemoveContainer" containerID="2df051703d6bcdb36340d0e433b2ff22545fcb9342a1a4bcfa90e53c431c5720"
Oct 30 13:06:57.772719 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:06:57.772702    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df051703d6bcdb36340d0e433b2ff22545fcb9342a1a4bcfa90e53c431c5720\": container with ID starting with 2df051703d6bcdb36340d0e433b2ff22545fcb9342a1a4bcfa90e53c431c5720 not found: ID does not exist" containerID="2df051703d6bcdb36340d0e433b2ff22545fcb9342a1a4bcfa90e53c431c5720"
Oct 30 13:06:57.772802 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.772725    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df051703d6bcdb36340d0e433b2ff22545fcb9342a1a4bcfa90e53c431c5720"} err="failed to get container status \"2df051703d6bcdb36340d0e433b2ff22545fcb9342a1a4bcfa90e53c431c5720\": rpc error: code = NotFound desc = could not find container \"2df051703d6bcdb36340d0e433b2ff22545fcb9342a1a4bcfa90e53c431c5720\": container with ID starting with 2df051703d6bcdb36340d0e433b2ff22545fcb9342a1a4bcfa90e53c431c5720 not found: ID does not exist"
Oct 30 13:06:57.814075 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.814053    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"]
Oct 30 13:06:57.819040 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:06:57.819016    4522 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186f443b_4af4_40dd_ad37_71be0c18ebf4.slice/crio-25fac7042e8940ec10d9caaa337a97887d64ca187d8d621f0a7eaceb1ec14615\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186f443b_4af4_40dd_ad37_71be0c18ebf4.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:57.825832 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:57.825811    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c81f42-dfcd4dcd7-kg544"]
Oct 30 13:06:58.333883 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:58.333850    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186f443b-4af4-40dd-ad37-71be0c18ebf4" path="/var/lib/kubelet/pods/186f443b-4af4-40dd-ad37-71be0c18ebf4/volumes"
Oct 30 13:06:59.121341 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:06:59.121311    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:00.711257 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:00.711215    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:00.815609 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:00.815572    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:00.920118 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:00.920024    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:01.355378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.348763    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 13:07:01.507489 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.507442    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" containerID="cri-o://20faa40ab9e6eb37d16905a0eda79784f0bad2ad8ca0fdabe3a56b2e2ce3fcbc" gracePeriod=45
Oct 30 13:07:01.545543 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.545510    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": dial tcp 10.133.0.80:8000: connect: connection refused"
Oct 30 13:07:01.805528 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.805497    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:07:01.832469 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.832438    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-home\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-home\") pod \"91ef2bf6-d815-459d-8162-2dc297e01897\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") "
Oct 30 13:07:01.832623 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.832550    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91ef2bf6-d815-459d-8162-2dc297e01897-tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef2bf6-d815-459d-8162-2dc297e01897-tls-certs\") pod \"91ef2bf6-d815-459d-8162-2dc297e01897\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") "
Oct 30 13:07:01.832695 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.832673    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-home" pod "91ef2bf6-d815-459d-8162-2dc297e01897" (UID: "91ef2bf6-d815-459d-8162-2dc297e01897"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:01.832760 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.832720    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-tmp-dir\") pod \"91ef2bf6-d815-459d-8162-2dc297e01897\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") "
Oct 30 13:07:01.832820 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.832772    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ef2bf6-d815-459d-8162-2dc297e01897-kube-api-access-9sb2b\" (UniqueName: \"kubernetes.io/projected/91ef2bf6-d815-459d-8162-2dc297e01897-kube-api-access-9sb2b\") pod \"91ef2bf6-d815-459d-8162-2dc297e01897\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") "
Oct 30 13:07:01.833170 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.832905    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-dshm\") pod \"91ef2bf6-d815-459d-8162-2dc297e01897\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") "
Oct 30 13:07:01.833737 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.832913    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-tmp-dir" pod "91ef2bf6-d815-459d-8162-2dc297e01897" (UID: "91ef2bf6-d815-459d-8162-2dc297e01897"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:01.833851 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.833791    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-model-cache\") pod \"91ef2bf6-d815-459d-8162-2dc297e01897\" (UID: \"91ef2bf6-d815-459d-8162-2dc297e01897\") "
Oct 30 13:07:01.834173 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.834112    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-model-cache" pod "91ef2bf6-d815-459d-8162-2dc297e01897" (UID: "91ef2bf6-d815-459d-8162-2dc297e01897"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:01.834748 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.834671    4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-model-cache\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:01.834748 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.834697    4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-home\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:01.834748 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.834710    4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-tmp-dir\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:01.836342 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.836314    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-dshm" pod "91ef2bf6-d815-459d-8162-2dc297e01897" (UID: "91ef2bf6-d815-459d-8162-2dc297e01897"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:01.836780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.836741    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ef2bf6-d815-459d-8162-2dc297e01897-tls-certs" pod "91ef2bf6-d815-459d-8162-2dc297e01897" (UID: "91ef2bf6-d815-459d-8162-2dc297e01897"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:07:01.842415 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.842389    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ef2bf6-d815-459d-8162-2dc297e01897-kube-api-access-9sb2b" pod "91ef2bf6-d815-459d-8162-2dc297e01897" (UID: "91ef2bf6-d815-459d-8162-2dc297e01897"). InnerVolumeSpecName "kube-api-access-9sb2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:01.845611 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.845587    4522 generic.go:356] "Generic (PLEG): container finished" podID="91ef2bf6-d815-459d-8162-2dc297e01897" containerID="20faa40ab9e6eb37d16905a0eda79784f0bad2ad8ca0fdabe3a56b2e2ce3fcbc" exitCode=0
Oct 30 13:07:01.845855 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.845760    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" event={"ID":"91ef2bf6-d815-459d-8162-2dc297e01897","Type":"ContainerDied","Data":"20faa40ab9e6eb37d16905a0eda79784f0bad2ad8ca0fdabe3a56b2e2ce3fcbc"}
Oct 30 13:07:01.845855 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.845791    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw" event={"ID":"91ef2bf6-d815-459d-8162-2dc297e01897","Type":"ContainerDied","Data":"5453de70d9c33026da98ee10008a882e4573f0a026a6e5364ac41ad02bf7d7d8"}
Oct 30 13:07:01.845855 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.845807    4522 scope.go:122] "RemoveContainer" containerID="20faa40ab9e6eb37d16905a0eda79784f0bad2ad8ca0fdabe3a56b2e2ce3fcbc"
Oct 30 13:07:01.846052 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.845907    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"
Oct 30 13:07:01.887400 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.883674    4522 scope.go:122] "RemoveContainer" containerID="20faa40ab9e6eb37d16905a0eda79784f0bad2ad8ca0fdabe3a56b2e2ce3fcbc"
Oct 30 13:07:01.887400 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:07:01.884072    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20faa40ab9e6eb37d16905a0eda79784f0bad2ad8ca0fdabe3a56b2e2ce3fcbc\": container with ID starting with 20faa40ab9e6eb37d16905a0eda79784f0bad2ad8ca0fdabe3a56b2e2ce3fcbc not found: ID does not exist" containerID="20faa40ab9e6eb37d16905a0eda79784f0bad2ad8ca0fdabe3a56b2e2ce3fcbc"
Oct 30 13:07:01.887400 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.884104    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20faa40ab9e6eb37d16905a0eda79784f0bad2ad8ca0fdabe3a56b2e2ce3fcbc"} err="failed to get container status \"20faa40ab9e6eb37d16905a0eda79784f0bad2ad8ca0fdabe3a56b2e2ce3fcbc\": rpc error: code = NotFound desc = could not find container \"20faa40ab9e6eb37d16905a0eda79784f0bad2ad8ca0fdabe3a56b2e2ce3fcbc\": container with ID starting with 20faa40ab9e6eb37d16905a0eda79784f0bad2ad8ca0fdabe3a56b2e2ce3fcbc not found: ID does not exist"
Oct 30 13:07:01.936317 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.936282    4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef2bf6-d815-459d-8162-2dc297e01897-tls-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:01.936317 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.936319    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sb2b\" (UniqueName: \"kubernetes.io/projected/91ef2bf6-d815-459d-8162-2dc297e01897-kube-api-access-9sb2b\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:01.936549 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.936333    4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef2bf6-d815-459d-8162-2dc297e01897-dshm\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:01.944125 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.944096    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"]
Oct 30 13:07:01.947714 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:01.947691    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4cd3fb/distinct-w0-4cd3fb-kserve-76799d77df-k5tpw"]
Oct 30 13:07:02.336759 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:02.336727    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" path="/var/lib/kubelet/pods/91ef2bf6-d815-459d-8162-2dc297e01897/volumes"
Oct 30 13:07:02.984438 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:02.984399    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:05.343435 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:05.343246    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:05.513383 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:05.491598    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:05.827803 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:05.827758    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:05.893378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:05.874421    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"]
Oct 30 13:07:05.893378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:05.874715    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy" containerID="cri-o://4bb9309b2a0bbbe917d52453399acf4edbd4c3981c09bddaf397c164e7bb4916" gracePeriod=30
Oct 30 13:07:06.024444 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:06.024407    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"]
Oct 30 13:07:06.024972 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:06.024935    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy" containerID="cri-o://acd717be8d37c52f00bd759a11b599d464c9623603dbde4fe1780e41b0ab1b37" gracePeriod=30
Oct 30 13:07:06.070373 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:06.069379    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"]
Oct 30 13:07:06.070373 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:06.069689    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerName="istio-proxy" containerID="cri-o://df8a2b662fd63696c6b94c72696f0ef0257670be97ae060b3b6490bf891c3d38" gracePeriod=30
Oct 30 13:07:06.070373 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:06.069945    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:06.081492 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:06.081413    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"]
Oct 30 13:07:06.081954 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:06.081904    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerName="istio-proxy" containerID="cri-o://3831989687ff66bde49bde3408b4c3c0a80cc8d1f4329273e8ff6f5c44b351f8" gracePeriod=30
Oct 30 13:07:07.727712 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:07.712428    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded"
Oct 30 13:07:08.067683 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:08.067642    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:08.843696 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:08.843654    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:10.361668 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:10.361637    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:10.888801 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:10.888766    4522 patch_prober.go:29] interesting pod/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:07:10.889120 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:10.889090    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:07:11.078055 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.078022    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:11.240496 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.240461    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:11.274411 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.272928    4522 generic.go:356] "Generic (PLEG): container finished" podID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerID="3831989687ff66bde49bde3408b4c3c0a80cc8d1f4329273e8ff6f5c44b351f8" exitCode=0
Oct 30 13:07:11.274411 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.273091    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" event={"ID":"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1","Type":"ContainerDied","Data":"3831989687ff66bde49bde3408b4c3c0a80cc8d1f4329273e8ff6f5c44b351f8"}
Oct 30 13:07:11.275309 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.275284    4522 generic.go:356] "Generic (PLEG): container finished" podID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerID="4bb9309b2a0bbbe917d52453399acf4edbd4c3981c09bddaf397c164e7bb4916" exitCode=0
Oct 30 13:07:11.275469 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.275441    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" event={"ID":"e726d2a8-8f05-471d-a7d1-2fa2c8df6550","Type":"ContainerDied","Data":"4bb9309b2a0bbbe917d52453399acf4edbd4c3981c09bddaf397c164e7bb4916"}
Oct 30 13:07:11.275531 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.275494    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql" event={"ID":"e726d2a8-8f05-471d-a7d1-2fa2c8df6550","Type":"ContainerDied","Data":"a1f5601aafb7deeee83f048e75f3359a13417e9a3fdb75f555520dab0f14ba76"}
Oct 30 13:07:11.275531 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.275520    4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1f5601aafb7deeee83f048e75f3359a13417e9a3fdb75f555520dab0f14ba76"
Oct 30 13:07:11.275634 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.275538    4522 scope.go:122] "RemoveContainer" containerID="21dac7f2de25108ea47a5b9d35144a09b50a83a933a16cbaebbaf5fdf305bb2a"
Oct 30 13:07:11.288483 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.288461    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 13:07:11.383867 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.383559    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:11.406418 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.404331    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.466985    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istiod-ca-cert\") pod \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467039    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-data\") pod \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467099    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-token\" (UniqueName: \"kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-token\") pod \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467190    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-certs\") pod \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467247    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-kube-api-access-nctz2\" (UniqueName: \"kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-kube-api-access-nctz2\") pod \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467279    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istiod-ca-cert\") pod \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467320    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-envoy\") pod \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467393    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-data\") pod \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467428    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-envoy\") pod \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467494    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-kube-api-access-wcgmf\" (UniqueName: \"kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-kube-api-access-wcgmf\") pod \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467537    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-socket\") pod \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467565    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-podinfo\") pod \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467599    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-credential-socket\") pod \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467669    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-podinfo\") pod \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467698    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-socket\") pod \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467730    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-credential-socket\") pod \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467781    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-token\" (UniqueName: \"kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-token\") pod \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\" (UID: \"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.467832    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-certs\") pod \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\" (UID: \"e726d2a8-8f05-471d-a7d1-2fa2c8df6550\") "
Oct 30 13:07:11.492375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.489632    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-socket" pod "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" (UID: "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.503554 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.502438    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-credential-socket" pod "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" (UID: "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.514435 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.506538    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-credential-socket" pod "e726d2a8-8f05-471d-a7d1-2fa2c8df6550" (UID: "e726d2a8-8f05-471d-a7d1-2fa2c8df6550"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.522973 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.515176    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-socket" pod "e726d2a8-8f05-471d-a7d1-2fa2c8df6550" (UID: "e726d2a8-8f05-471d-a7d1-2fa2c8df6550"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.522973 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.515268    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-certs" pod "e726d2a8-8f05-471d-a7d1-2fa2c8df6550" (UID: "e726d2a8-8f05-471d-a7d1-2fa2c8df6550"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.525427 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.524702    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-certs" pod "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" (UID: "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.525427 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.525171    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istiod-ca-cert" pod "e726d2a8-8f05-471d-a7d1-2fa2c8df6550" (UID: "e726d2a8-8f05-471d-a7d1-2fa2c8df6550"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:11.525639 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.525479    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-data" pod "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" (UID: "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.525992 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.525965    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istiod-ca-cert" pod "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" (UID: "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:11.530171 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.528959    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-data" pod "e726d2a8-8f05-471d-a7d1-2fa2c8df6550" (UID: "e726d2a8-8f05-471d-a7d1-2fa2c8df6550"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.530554 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.530497    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-token" pod "e726d2a8-8f05-471d-a7d1-2fa2c8df6550" (UID: "e726d2a8-8f05-471d-a7d1-2fa2c8df6550"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:11.531100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.531077    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-envoy" pod "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" (UID: "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.532567 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.532533    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-kube-api-access-nctz2" pod "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" (UID: "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1"). InnerVolumeSpecName "kube-api-access-nctz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:11.534176 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.534140    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-podinfo" pod "e726d2a8-8f05-471d-a7d1-2fa2c8df6550" (UID: "e726d2a8-8f05-471d-a7d1-2fa2c8df6550"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:11.535864 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.535836    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-token" pod "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" (UID: "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:11.536131 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.536093    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-kube-api-access-wcgmf" pod "e726d2a8-8f05-471d-a7d1-2fa2c8df6550" (UID: "e726d2a8-8f05-471d-a7d1-2fa2c8df6550"). InnerVolumeSpecName "kube-api-access-wcgmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:11.536392 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.536327    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-envoy" pod "e726d2a8-8f05-471d-a7d1-2fa2c8df6550" (UID: "e726d2a8-8f05-471d-a7d1-2fa2c8df6550"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.537582 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.537546    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-podinfo" pod "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" (UID: "e6aa45d0-5cee-463c-a2d4-89ee83f50aa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:11.570285 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570260    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570291    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nctz2\" (UniqueName: \"kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-kube-api-access-nctz2\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570308    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570320    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570333    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570345    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570372    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcgmf\" (UniqueName: \"kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-kube-api-access-wcgmf\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570389    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570404    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570683 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570418    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570683 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570432    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570683 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570447    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570683 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570460    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570683 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570473    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570683 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570486    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570683 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570499    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570683 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570510    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.570683 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.570522    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e726d2a8-8f05-471d-a7d1-2fa2c8df6550-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.667304 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.667281    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 13:07:11.671512 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.671490    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-kube-api-access-2qzs7\" (UniqueName: \"kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-kube-api-access-2qzs7\") pod \"208f0cbd-3ed2-4e90-9929-277352e907d3\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") "
Oct 30 13:07:11.671614 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.671525    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-data\") pod \"208f0cbd-3ed2-4e90-9929-277352e907d3\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") "
Oct 30 13:07:11.671614 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.671574    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-certs\") pod \"208f0cbd-3ed2-4e90-9929-277352e907d3\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") "
Oct 30 13:07:11.671614 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.671606    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208f0cbd-3ed2-4e90-9929-277352e907d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208f0cbd-3ed2-4e90-9929-277352e907d3-istiod-ca-cert\") pod \"208f0cbd-3ed2-4e90-9929-277352e907d3\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") "
Oct 30 13:07:11.671827 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.671662    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-podinfo\") pod \"208f0cbd-3ed2-4e90-9929-277352e907d3\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") "
Oct 30 13:07:11.671827 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.671697    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-envoy\") pod \"208f0cbd-3ed2-4e90-9929-277352e907d3\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") "
Oct 30 13:07:11.671827 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.671755    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-token\" (UniqueName: \"kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-token\") pod \"208f0cbd-3ed2-4e90-9929-277352e907d3\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") "
Oct 30 13:07:11.671995 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.671848    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-certs" pod "208f0cbd-3ed2-4e90-9929-277352e907d3" (UID: "208f0cbd-3ed2-4e90-9929-277352e907d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.671995 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.671864    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-credential-socket\") pod \"208f0cbd-3ed2-4e90-9929-277352e907d3\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") "
Oct 30 13:07:11.671995 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.671967    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-socket\") pod \"208f0cbd-3ed2-4e90-9929-277352e907d3\" (UID: \"208f0cbd-3ed2-4e90-9929-277352e907d3\") "
Oct 30 13:07:11.672143 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.672047    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208f0cbd-3ed2-4e90-9929-277352e907d3-istiod-ca-cert" pod "208f0cbd-3ed2-4e90-9929-277352e907d3" (UID: "208f0cbd-3ed2-4e90-9929-277352e907d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:11.672143 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.672068    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-credential-socket" pod "208f0cbd-3ed2-4e90-9929-277352e907d3" (UID: "208f0cbd-3ed2-4e90-9929-277352e907d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.672143 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.672094    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-data" pod "208f0cbd-3ed2-4e90-9929-277352e907d3" (UID: "208f0cbd-3ed2-4e90-9929-277352e907d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.672325 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.672309    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-socket" pod "208f0cbd-3ed2-4e90-9929-277352e907d3" (UID: "208f0cbd-3ed2-4e90-9929-277352e907d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.672609 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.672586    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.672650 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.672617    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.672650 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.672633    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.672650 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.672646    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.672767 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.672660    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208f0cbd-3ed2-4e90-9929-277352e907d3-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.673960 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.673933    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-kube-api-access-2qzs7" pod "208f0cbd-3ed2-4e90-9929-277352e907d3" (UID: "208f0cbd-3ed2-4e90-9929-277352e907d3"). InnerVolumeSpecName "kube-api-access-2qzs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:11.674732 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.674712    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-token" pod "208f0cbd-3ed2-4e90-9929-277352e907d3" (UID: "208f0cbd-3ed2-4e90-9929-277352e907d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:11.675163 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.675143    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-envoy" pod "208f0cbd-3ed2-4e90-9929-277352e907d3" (UID: "208f0cbd-3ed2-4e90-9929-277352e907d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.675267 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.675252    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-podinfo" pod "208f0cbd-3ed2-4e90-9929-277352e907d3" (UID: "208f0cbd-3ed2-4e90-9929-277352e907d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:11.684394 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.684374    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 13:07:11.774064 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.774035    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-kube-api-access-w7qnv\" (UniqueName: \"kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-kube-api-access-w7qnv\") pod \"f2019baf-7b8e-4c05-96b8-b48f93809985\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") "
Oct 30 13:07:11.774187 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.774130    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-credential-socket\") pod \"f2019baf-7b8e-4c05-96b8-b48f93809985\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") "
Oct 30 13:07:11.774240 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.774191    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-certs\") pod \"f2019baf-7b8e-4c05-96b8-b48f93809985\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") "
Oct 30 13:07:11.774240 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.774220    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-data\") pod \"f2019baf-7b8e-4c05-96b8-b48f93809985\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") "
Oct 30 13:07:11.774339 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.774291    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-envoy\") pod \"f2019baf-7b8e-4c05-96b8-b48f93809985\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") "
Oct 30 13:07:11.774420 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.774384    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-podinfo\") pod \"f2019baf-7b8e-4c05-96b8-b48f93809985\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") "
Oct 30 13:07:11.774483 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.774444    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2019baf-7b8e-4c05-96b8-b48f93809985-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2019baf-7b8e-4c05-96b8-b48f93809985-istiod-ca-cert\") pod \"f2019baf-7b8e-4c05-96b8-b48f93809985\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") "
Oct 30 13:07:11.774541 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.774491    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-credential-socket" pod "f2019baf-7b8e-4c05-96b8-b48f93809985" (UID: "f2019baf-7b8e-4c05-96b8-b48f93809985"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.774600 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.774541    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-socket\") pod \"f2019baf-7b8e-4c05-96b8-b48f93809985\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") "
Oct 30 13:07:11.774600 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.774585    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-token\" (UniqueName: \"kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-token\") pod \"f2019baf-7b8e-4c05-96b8-b48f93809985\" (UID: \"f2019baf-7b8e-4c05-96b8-b48f93809985\") "
Oct 30 13:07:11.775054 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.774535    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-certs" pod "f2019baf-7b8e-4c05-96b8-b48f93809985" (UID: "f2019baf-7b8e-4c05-96b8-b48f93809985"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.775150 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.774824    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-data" pod "f2019baf-7b8e-4c05-96b8-b48f93809985" (UID: "f2019baf-7b8e-4c05-96b8-b48f93809985"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.775150 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.774839    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2019baf-7b8e-4c05-96b8-b48f93809985-istiod-ca-cert" pod "f2019baf-7b8e-4c05-96b8-b48f93809985" (UID: "f2019baf-7b8e-4c05-96b8-b48f93809985"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:11.775150 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.775031    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-socket" pod "f2019baf-7b8e-4c05-96b8-b48f93809985" (UID: "f2019baf-7b8e-4c05-96b8-b48f93809985"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.775314 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.775292    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.775314 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.775311    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.775465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.775326    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2019baf-7b8e-4c05-96b8-b48f93809985-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.775465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.775340    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.775465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.775380    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qzs7\" (UniqueName: \"kubernetes.io/projected/208f0cbd-3ed2-4e90-9929-277352e907d3-kube-api-access-2qzs7\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.775465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.775395    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.775465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.775408    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.775465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.775422    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.775465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.775435    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208f0cbd-3ed2-4e90-9929-277352e907d3-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.776685 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.776664    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-kube-api-access-w7qnv" pod "f2019baf-7b8e-4c05-96b8-b48f93809985" (UID: "f2019baf-7b8e-4c05-96b8-b48f93809985"). InnerVolumeSpecName "kube-api-access-w7qnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:11.776836 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.776817    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-podinfo" pod "f2019baf-7b8e-4c05-96b8-b48f93809985" (UID: "f2019baf-7b8e-4c05-96b8-b48f93809985"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:11.777117 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.777100    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-token" pod "f2019baf-7b8e-4c05-96b8-b48f93809985" (UID: "f2019baf-7b8e-4c05-96b8-b48f93809985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:11.777184 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.777160    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-envoy" pod "f2019baf-7b8e-4c05-96b8-b48f93809985" (UID: "f2019baf-7b8e-4c05-96b8-b48f93809985"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:11.800429 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.800406    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:11.877006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.876984    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.877006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.877006    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7qnv\" (UniqueName: \"kubernetes.io/projected/f2019baf-7b8e-4c05-96b8-b48f93809985-kube-api-access-w7qnv\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.877133 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.877015    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:11.877133 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:11.877024    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2019baf-7b8e-4c05-96b8-b48f93809985-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.282607 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.282578    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"
Oct 30 13:07:12.282758 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.282607    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82" event={"ID":"e6aa45d0-5cee-463c-a2d4-89ee83f50aa1","Type":"ContainerDied","Data":"313d678e259ecdda19f984a9026fab2695f2b1fb79cb46ac43ff4080458d2a14"}
Oct 30 13:07:12.282758 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.282666    4522 scope.go:122] "RemoveContainer" containerID="3831989687ff66bde49bde3408b4c3c0a80cc8d1f4329273e8ff6f5c44b351f8"
Oct 30 13:07:12.284048 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.283972    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"
Oct 30 13:07:12.285564 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.285537    4522 generic.go:356] "Generic (PLEG): container finished" podID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerID="df8a2b662fd63696c6b94c72696f0ef0257670be97ae060b3b6490bf891c3d38" exitCode=0
Oct 30 13:07:12.285667 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.285577    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" event={"ID":"208f0cbd-3ed2-4e90-9929-277352e907d3","Type":"ContainerDied","Data":"df8a2b662fd63696c6b94c72696f0ef0257670be97ae060b3b6490bf891c3d38"}
Oct 30 13:07:12.285667 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.285632    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8" event={"ID":"208f0cbd-3ed2-4e90-9929-277352e907d3","Type":"ContainerDied","Data":"d59874f36e001efcde35fe39ebeb56c48e7b830f5daf5b826440a50ce6c521d2"}
Oct 30 13:07:12.285667 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.285592    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"
Oct 30 13:07:12.287682 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.287651    4522 generic.go:356] "Generic (PLEG): container finished" podID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerID="acd717be8d37c52f00bd759a11b599d464c9623603dbde4fe1780e41b0ab1b37" exitCode=0
Oct 30 13:07:12.287805 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.287717    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"
Oct 30 13:07:12.287805 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.287760    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" event={"ID":"f2019baf-7b8e-4c05-96b8-b48f93809985","Type":"ContainerDied","Data":"acd717be8d37c52f00bd759a11b599d464c9623603dbde4fe1780e41b0ab1b37"}
Oct 30 13:07:12.287924 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.287805    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2" event={"ID":"f2019baf-7b8e-4c05-96b8-b48f93809985","Type":"ContainerDied","Data":"aee61c8072e2631f897ea9a3e0e370731cf1b7a45ba6095ba4efab8c01cbfc2f"}
Oct 30 13:07:12.298951 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.298932    4522 scope.go:122] "RemoveContainer" containerID="df8a2b662fd63696c6b94c72696f0ef0257670be97ae060b3b6490bf891c3d38"
Oct 30 13:07:12.307159 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.307143    4522 scope.go:122] "RemoveContainer" containerID="df8a2b662fd63696c6b94c72696f0ef0257670be97ae060b3b6490bf891c3d38"
Oct 30 13:07:12.307424 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:07:12.307407    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8a2b662fd63696c6b94c72696f0ef0257670be97ae060b3b6490bf891c3d38\": container with ID starting with df8a2b662fd63696c6b94c72696f0ef0257670be97ae060b3b6490bf891c3d38 not found: ID does not exist" containerID="df8a2b662fd63696c6b94c72696f0ef0257670be97ae060b3b6490bf891c3d38"
Oct 30 13:07:12.307475 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.307430    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8a2b662fd63696c6b94c72696f0ef0257670be97ae060b3b6490bf891c3d38"} err="failed to get container status \"df8a2b662fd63696c6b94c72696f0ef0257670be97ae060b3b6490bf891c3d38\": rpc error: code = NotFound desc = could not find container \"df8a2b662fd63696c6b94c72696f0ef0257670be97ae060b3b6490bf891c3d38\": container with ID starting with df8a2b662fd63696c6b94c72696f0ef0257670be97ae060b3b6490bf891c3d38 not found: ID does not exist"
Oct 30 13:07:12.307475 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.307445    4522 scope.go:122] "RemoveContainer" containerID="acd717be8d37c52f00bd759a11b599d464c9623603dbde4fe1780e41b0ab1b37"
Oct 30 13:07:12.315306 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.315288    4522 scope.go:122] "RemoveContainer" containerID="acd717be8d37c52f00bd759a11b599d464c9623603dbde4fe1780e41b0ab1b37"
Oct 30 13:07:12.315607 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:07:12.315590    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd717be8d37c52f00bd759a11b599d464c9623603dbde4fe1780e41b0ab1b37\": container with ID starting with acd717be8d37c52f00bd759a11b599d464c9623603dbde4fe1780e41b0ab1b37 not found: ID does not exist" containerID="acd717be8d37c52f00bd759a11b599d464c9623603dbde4fe1780e41b0ab1b37"
Oct 30 13:07:12.315659 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.315612    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd717be8d37c52f00bd759a11b599d464c9623603dbde4fe1780e41b0ab1b37"} err="failed to get container status \"acd717be8d37c52f00bd759a11b599d464c9623603dbde4fe1780e41b0ab1b37\": rpc error: code = NotFound desc = could not find container \"acd717be8d37c52f00bd759a11b599d464c9623603dbde4fe1780e41b0ab1b37\": container with ID starting with acd717be8d37c52f00bd759a11b599d464c9623603dbde4fe1780e41b0ab1b37 not found: ID does not exist"
Oct 30 13:07:12.366953 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.366931    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"]
Oct 30 13:07:12.390662 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.390636    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-4lg82"]
Oct 30 13:07:12.435038 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.435014    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"]
Oct 30 13:07:12.447722 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.447702    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-869c8"]
Oct 30 13:07:12.558643 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.558592    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"]
Oct 30 13:07:12.564458 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.564436    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-ldbql"]
Oct 30 13:07:12.610126 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.610106    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"]
Oct 30 13:07:12.620049 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:12.620031    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c81f42-openshift-default-c7f9d447-6z2m2"]
Oct 30 13:07:14.071494 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:14.071457    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:14.334511 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:14.334440    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" path="/var/lib/kubelet/pods/208f0cbd-3ed2-4e90-9929-277352e907d3/volumes"
Oct 30 13:07:14.334911 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:14.334897    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" path="/var/lib/kubelet/pods/e6aa45d0-5cee-463c-a2d4-89ee83f50aa1/volumes"
Oct 30 13:07:14.335307 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:14.335294    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" path="/var/lib/kubelet/pods/e726d2a8-8f05-471d-a7d1-2fa2c8df6550/volumes"
Oct 30 13:07:14.336043 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:14.336028    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" path="/var/lib/kubelet/pods/f2019baf-7b8e-4c05-96b8-b48f93809985/volumes"
Oct 30 13:07:16.334743 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:16.334696    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:17.896063 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:17.896030    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:18.010912 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:18.010768    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:18.168947 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:18.168868    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:18.289445 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:18.288628    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"]
Oct 30 13:07:18.289445 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:18.288900    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy" containerID="cri-o://40f0e7b9f5e84eec9cc6d8e2030b9aecdb7f855c67aac71396e707f42c7d091b" gracePeriod=30
Oct 30 13:07:18.289445 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:18.288624    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:18.300322 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:18.300294    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"]
Oct 30 13:07:18.300560 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:18.300532    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy" containerID="cri-o://bd55715591d3d7111a83f1013c14e871c6990a14bc22f9e788a5bef9869b2c24" gracePeriod=30
Oct 30 13:07:18.311027 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:18.311003    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"]
Oct 30 13:07:18.311223 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:18.311200    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerName="istio-proxy" containerID="cri-o://d9a9f205b0574c462664d0ee7330b56607a3d7383f1cc945538c1a7beffdae68" gracePeriod=30
Oct 30 13:07:18.318131 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:18.318099    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"]
Oct 30 13:07:18.318338 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:18.318314    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy" containerID="cri-o://3547337f18ddfc64ee089a78195d25149a5bd7e4afe6c8cd67d89f51abbc0381" gracePeriod=30
Oct 30 13:07:19.982209 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:19.982175    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:21.134392 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:07:21.134332    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:07:21.809912 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:21.809871    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:23.175909 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.175869    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:23.268913 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.268870    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:23.583052 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.583030    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 13:07:23.610564 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.610540    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 13:07:23.622751 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.622724    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-data\") pod \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") "
Oct 30 13:07:23.622878 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.622813    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-certs\") pod \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") "
Oct 30 13:07:23.622943 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.622893    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-kube-api-access-st846\" (UniqueName: \"kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-kube-api-access-st846\") pod \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") "
Oct 30 13:07:23.623003 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.622972    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-token\" (UniqueName: \"kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-token\") pod \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") "
Oct 30 13:07:23.623057 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.623041    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-socket\") pod \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") "
Oct 30 13:07:23.623131 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.623113    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-envoy\") pod \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") "
Oct 30 13:07:23.623214 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.623180    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-certs" pod "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" (UID: "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.623214 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.623185    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istiod-ca-cert\") pod \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") "
Oct 30 13:07:23.623404 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.623275    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-credential-socket\") pod \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") "
Oct 30 13:07:23.623503 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.623489    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istiod-ca-cert" pod "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" (UID: "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:23.623587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.623574    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-podinfo\") pod \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\" (UID: \"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6\") "
Oct 30 13:07:23.624038 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.624017    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.624151 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.624039    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.624419 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.624329    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-data" pod "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" (UID: "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.624601 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.624566    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-credential-socket" pod "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" (UID: "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.624780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.624743    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-socket" pod "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" (UID: "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.625730 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.625706    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-token" pod "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" (UID: "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.626493 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.625777    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-kube-api-access-st846" pod "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" (UID: "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6"). InnerVolumeSpecName "kube-api-access-st846". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.626770 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.626748    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-podinfo" pod "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" (UID: "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:23.627488 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.627467    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-envoy" pod "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" (UID: "fa8a14d3-f3ed-43ec-aed2-8017b4f195f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.688200 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.688175    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 13:07:23.706482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.706455    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 13:07:23.725392 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725342    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-kube-api-access-k9kds\" (UniqueName: \"kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-kube-api-access-k9kds\") pod \"14b9e4de-979a-4e21-9714-c6a99f26d861\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") "
Oct 30 13:07:23.725567 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725431    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/674b4304-a144-449c-9c7b-bd026b669f74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674b4304-a144-449c-9c7b-bd026b669f74-istio-podinfo\") pod \"674b4304-a144-449c-9c7b-bd026b669f74\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") "
Oct 30 13:07:23.725567 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725479    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-certs\") pod \"14b9e4de-979a-4e21-9714-c6a99f26d861\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") "
Oct 30 13:07:23.725567 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725541    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-istio-token\" (UniqueName: \"kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-istio-token\") pod \"674b4304-a144-449c-9c7b-bd026b669f74\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") "
Oct 30 13:07:23.725733 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725591    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-socket\") pod \"674b4304-a144-449c-9c7b-bd026b669f74\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") "
Oct 30 13:07:23.725733 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725637    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-kube-api-access-vfrcr\" (UniqueName: \"kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-kube-api-access-vfrcr\") pod \"674b4304-a144-449c-9c7b-bd026b669f74\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") "
Oct 30 13:07:23.725733 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725667    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b9e4de-979a-4e21-9714-c6a99f26d861-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b9e4de-979a-4e21-9714-c6a99f26d861-istiod-ca-cert\") pod \"14b9e4de-979a-4e21-9714-c6a99f26d861\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") "
Oct 30 13:07:23.725733 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725709    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-token\" (UniqueName: \"kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-token\") pod \"14b9e4de-979a-4e21-9714-c6a99f26d861\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") "
Oct 30 13:07:23.725937 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725766    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-certs\") pod \"674b4304-a144-449c-9c7b-bd026b669f74\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") "
Oct 30 13:07:23.725937 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725798    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-data\") pod \"14b9e4de-979a-4e21-9714-c6a99f26d861\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") "
Oct 30 13:07:23.725937 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725837    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-envoy\") pod \"14b9e4de-979a-4e21-9714-c6a99f26d861\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") "
Oct 30 13:07:23.725937 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725903    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-credential-socket\") pod \"14b9e4de-979a-4e21-9714-c6a99f26d861\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") "
Oct 30 13:07:23.726145 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725944    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-credential-socket\") pod \"674b4304-a144-449c-9c7b-bd026b669f74\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") "
Oct 30 13:07:23.726145 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.725973    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-socket\") pod \"14b9e4de-979a-4e21-9714-c6a99f26d861\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") "
Oct 30 13:07:23.726145 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726002    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-envoy\") pod \"674b4304-a144-449c-9c7b-bd026b669f74\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") "
Oct 30 13:07:23.726145 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726079    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/674b4304-a144-449c-9c7b-bd026b669f74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674b4304-a144-449c-9c7b-bd026b669f74-istiod-ca-cert\") pod \"674b4304-a144-449c-9c7b-bd026b669f74\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") "
Oct 30 13:07:23.726145 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726130    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-data\") pod \"674b4304-a144-449c-9c7b-bd026b669f74\" (UID: \"674b4304-a144-449c-9c7b-bd026b669f74\") "
Oct 30 13:07:23.726145 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726127    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-certs" pod "14b9e4de-979a-4e21-9714-c6a99f26d861" (UID: "14b9e4de-979a-4e21-9714-c6a99f26d861"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.726475 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726161    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-podinfo\") pod \"14b9e4de-979a-4e21-9714-c6a99f26d861\" (UID: \"14b9e4de-979a-4e21-9714-c6a99f26d861\") "
Oct 30 13:07:23.726475 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726208    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-socket" pod "674b4304-a144-449c-9c7b-bd026b669f74" (UID: "674b4304-a144-449c-9c7b-bd026b669f74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.726475 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726374    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-credential-socket" pod "674b4304-a144-449c-9c7b-bd026b669f74" (UID: "674b4304-a144-449c-9c7b-bd026b669f74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.726651 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726485    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-data" pod "14b9e4de-979a-4e21-9714-c6a99f26d861" (UID: "14b9e4de-979a-4e21-9714-c6a99f26d861"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.726801 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726778    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.726881 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726808    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.726881 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726824    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.726881 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726839    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.726881 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726852    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.726881 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726873    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st846\" (UniqueName: \"kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-kube-api-access-st846\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.727148 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726887    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.727148 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726901    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.727148 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726914    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.727148 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726927    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.727148 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.726939    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.727564 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.727153    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-certs" pod "674b4304-a144-449c-9c7b-bd026b669f74" (UID: "674b4304-a144-449c-9c7b-bd026b669f74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.727628 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.727562    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-data" pod "674b4304-a144-449c-9c7b-bd026b669f74" (UID: "674b4304-a144-449c-9c7b-bd026b669f74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.727628 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.727568    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-credential-socket" pod "14b9e4de-979a-4e21-9714-c6a99f26d861" (UID: "14b9e4de-979a-4e21-9714-c6a99f26d861"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.727742 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.727704    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-socket" pod "14b9e4de-979a-4e21-9714-c6a99f26d861" (UID: "14b9e4de-979a-4e21-9714-c6a99f26d861"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.727797 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.727759    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b9e4de-979a-4e21-9714-c6a99f26d861-istiod-ca-cert" pod "14b9e4de-979a-4e21-9714-c6a99f26d861" (UID: "14b9e4de-979a-4e21-9714-c6a99f26d861"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:23.727856 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.727844    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674b4304-a144-449c-9c7b-bd026b669f74-istiod-ca-cert" pod "674b4304-a144-449c-9c7b-bd026b669f74" (UID: "674b4304-a144-449c-9c7b-bd026b669f74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:23.727962 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.727939    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/674b4304-a144-449c-9c7b-bd026b669f74-istio-podinfo" pod "674b4304-a144-449c-9c7b-bd026b669f74" (UID: "674b4304-a144-449c-9c7b-bd026b669f74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:23.728029 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.727981    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-kube-api-access-k9kds" pod "14b9e4de-979a-4e21-9714-c6a99f26d861" (UID: "14b9e4de-979a-4e21-9714-c6a99f26d861"). InnerVolumeSpecName "kube-api-access-k9kds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.728831 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.728796    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-podinfo" pod "14b9e4de-979a-4e21-9714-c6a99f26d861" (UID: "14b9e4de-979a-4e21-9714-c6a99f26d861"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:23.729918 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.729834    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-token" pod "14b9e4de-979a-4e21-9714-c6a99f26d861" (UID: "14b9e4de-979a-4e21-9714-c6a99f26d861"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.729918 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.729901    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-envoy" pod "14b9e4de-979a-4e21-9714-c6a99f26d861" (UID: "14b9e4de-979a-4e21-9714-c6a99f26d861"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.730038 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.729945    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-kube-api-access-vfrcr" pod "674b4304-a144-449c-9c7b-bd026b669f74" (UID: "674b4304-a144-449c-9c7b-bd026b669f74"). InnerVolumeSpecName "kube-api-access-vfrcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.730160 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.730133    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-istio-token" pod "674b4304-a144-449c-9c7b-bd026b669f74" (UID: "674b4304-a144-449c-9c7b-bd026b669f74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.730874 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.730846    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-envoy" pod "674b4304-a144-449c-9c7b-bd026b669f74" (UID: "674b4304-a144-449c-9c7b-bd026b669f74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.743625 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.743595    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:23.828787 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.828285    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-podinfo\") pod \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") "
Oct 30 13:07:23.828787 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.828393    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-kube-api-access-bwt6q\" (UniqueName: \"kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-kube-api-access-bwt6q\") pod \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") "
Oct 30 13:07:23.828787 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.828635    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-token\" (UniqueName: \"kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-token\") pod \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") "
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.830284    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-socket\") pod \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") "
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.830386    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-envoy\") pod \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") "
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.830424    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istiod-ca-cert\") pod \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") "
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.830472    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-certs\") pod \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") "
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.830582    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-credential-socket\") pod \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") "
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.830630    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-data\") pod \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\" (UID: \"8b94e06d-f102-43e5-86f2-ab0c2065b4ca\") "
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831214    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674b4304-a144-449c-9c7b-bd026b669f74-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831237    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831251    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831265    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9kds\" (UniqueName: \"kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-kube-api-access-k9kds\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831279    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674b4304-a144-449c-9c7b-bd026b669f74-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831292    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831307    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfrcr\" (UniqueName: \"kubernetes.io/projected/674b4304-a144-449c-9c7b-bd026b669f74-kube-api-access-vfrcr\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831320    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b9e4de-979a-4e21-9714-c6a99f26d861-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831333    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831344    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831373    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831386    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831399    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b9e4de-979a-4e21-9714-c6a99f26d861-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831412    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674b4304-a144-449c-9c7b-bd026b669f74-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831679    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-data" pod "8b94e06d-f102-43e5-86f2-ab0c2065b4ca" (UID: "8b94e06d-f102-43e5-86f2-ab0c2065b4ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.832396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.831874    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-socket" pod "8b94e06d-f102-43e5-86f2-ab0c2065b4ca" (UID: "8b94e06d-f102-43e5-86f2-ab0c2065b4ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.838183 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.837863    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-token" pod "8b94e06d-f102-43e5-86f2-ab0c2065b4ca" (UID: "8b94e06d-f102-43e5-86f2-ab0c2065b4ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.838491 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.838388    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-certs" pod "8b94e06d-f102-43e5-86f2-ab0c2065b4ca" (UID: "8b94e06d-f102-43e5-86f2-ab0c2065b4ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.839511 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.838755    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-podinfo" pod "8b94e06d-f102-43e5-86f2-ab0c2065b4ca" (UID: "8b94e06d-f102-43e5-86f2-ab0c2065b4ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:23.839511 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.838966    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-credential-socket" pod "8b94e06d-f102-43e5-86f2-ab0c2065b4ca" (UID: "8b94e06d-f102-43e5-86f2-ab0c2065b4ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.839511 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.839420    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istiod-ca-cert" pod "8b94e06d-f102-43e5-86f2-ab0c2065b4ca" (UID: "8b94e06d-f102-43e5-86f2-ab0c2065b4ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:23.839890 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.839856    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-kube-api-access-bwt6q" pod "8b94e06d-f102-43e5-86f2-ab0c2065b4ca" (UID: "8b94e06d-f102-43e5-86f2-ab0c2065b4ca"). InnerVolumeSpecName "kube-api-access-bwt6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:23.841214 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.841187    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-envoy" pod "8b94e06d-f102-43e5-86f2-ab0c2065b4ca" (UID: "8b94e06d-f102-43e5-86f2-ab0c2065b4ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:23.933184 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.933120    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.933184 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.933150    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.933184 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.933165    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.933184 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.933177    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.933462 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.933190    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwt6q\" (UniqueName: \"kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-kube-api-access-bwt6q\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.933462 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.933202    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.933462 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.933215    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.933462 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.933227    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:23.933462 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:23.933239    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b94e06d-f102-43e5-86f2-ab0c2065b4ca-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:07:24.360183 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.360047    4522 generic.go:356] "Generic (PLEG): container finished" podID="674b4304-a144-449c-9c7b-bd026b669f74" containerID="3547337f18ddfc64ee089a78195d25149a5bd7e4afe6c8cd67d89f51abbc0381" exitCode=0
Oct 30 13:07:24.360183 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.360082    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" event={"ID":"674b4304-a144-449c-9c7b-bd026b669f74","Type":"ContainerDied","Data":"3547337f18ddfc64ee089a78195d25149a5bd7e4afe6c8cd67d89f51abbc0381"}
Oct 30 13:07:24.360183 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.360123    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"
Oct 30 13:07:24.360183 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.360133    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2" event={"ID":"674b4304-a144-449c-9c7b-bd026b669f74","Type":"ContainerDied","Data":"a38759b92adbc4489062845766c5778bcc45c8bbce6a7c24071d71a3a736341a"}
Oct 30 13:07:24.360183 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.360164    4522 scope.go:122] "RemoveContainer" containerID="3547337f18ddfc64ee089a78195d25149a5bd7e4afe6c8cd67d89f51abbc0381"
Oct 30 13:07:24.361604 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.361579    4522 generic.go:356] "Generic (PLEG): container finished" podID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerID="bd55715591d3d7111a83f1013c14e871c6990a14bc22f9e788a5bef9869b2c24" exitCode=0
Oct 30 13:07:24.361742 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.361625    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"
Oct 30 13:07:24.361742 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.361666    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" event={"ID":"14b9e4de-979a-4e21-9714-c6a99f26d861","Type":"ContainerDied","Data":"bd55715591d3d7111a83f1013c14e871c6990a14bc22f9e788a5bef9869b2c24"}
Oct 30 13:07:24.361742 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.361716    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4" event={"ID":"14b9e4de-979a-4e21-9714-c6a99f26d861","Type":"ContainerDied","Data":"bdbcebfe761ec7ac9ef0994dda11e89009aa9bb79d0447f045c6afa28b6b1e83"}
Oct 30 13:07:24.363641 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.363615    4522 generic.go:356] "Generic (PLEG): container finished" podID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerID="d9a9f205b0574c462664d0ee7330b56607a3d7383f1cc945538c1a7beffdae68" exitCode=0
Oct 30 13:07:24.363743 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.363637    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" event={"ID":"8b94e06d-f102-43e5-86f2-ab0c2065b4ca","Type":"ContainerDied","Data":"d9a9f205b0574c462664d0ee7330b56607a3d7383f1cc945538c1a7beffdae68"}
Oct 30 13:07:24.363743 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.363683    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj" event={"ID":"8b94e06d-f102-43e5-86f2-ab0c2065b4ca","Type":"ContainerDied","Data":"9a6c57a9794bc3838e256d7c7322a1cc2f0a79f37768e35f8cd5374068a259a8"}
Oct 30 13:07:24.363743 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.363692    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"
Oct 30 13:07:24.374609 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.374582    4522 generic.go:356] "Generic (PLEG): container finished" podID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerID="40f0e7b9f5e84eec9cc6d8e2030b9aecdb7f855c67aac71396e707f42c7d091b" exitCode=0
Oct 30 13:07:24.374788 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.374619    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" event={"ID":"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6","Type":"ContainerDied","Data":"40f0e7b9f5e84eec9cc6d8e2030b9aecdb7f855c67aac71396e707f42c7d091b"}
Oct 30 13:07:24.374904 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.374845    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v" event={"ID":"fa8a14d3-f3ed-43ec-aed2-8017b4f195f6","Type":"ContainerDied","Data":"0fe59e0cc347c43b768d248777a98b344ca6555aba3259b5dda70c92004473ff"}
Oct 30 13:07:24.374904 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.374669    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"
Oct 30 13:07:24.381250 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.381228    4522 scope.go:122] "RemoveContainer" containerID="f3c2d524d044dbc6913cdf84b7a740c31e43c2ccab9abef1f5deefd076ef2cef"
Oct 30 13:07:24.389754 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.389735    4522 scope.go:122] "RemoveContainer" containerID="3547337f18ddfc64ee089a78195d25149a5bd7e4afe6c8cd67d89f51abbc0381"
Oct 30 13:07:24.390010 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:07:24.389989    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3547337f18ddfc64ee089a78195d25149a5bd7e4afe6c8cd67d89f51abbc0381\": container with ID starting with 3547337f18ddfc64ee089a78195d25149a5bd7e4afe6c8cd67d89f51abbc0381 not found: ID does not exist" containerID="3547337f18ddfc64ee089a78195d25149a5bd7e4afe6c8cd67d89f51abbc0381"
Oct 30 13:07:24.390097 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.390021    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3547337f18ddfc64ee089a78195d25149a5bd7e4afe6c8cd67d89f51abbc0381"} err="failed to get container status \"3547337f18ddfc64ee089a78195d25149a5bd7e4afe6c8cd67d89f51abbc0381\": rpc error: code = NotFound desc = could not find container \"3547337f18ddfc64ee089a78195d25149a5bd7e4afe6c8cd67d89f51abbc0381\": container with ID starting with 3547337f18ddfc64ee089a78195d25149a5bd7e4afe6c8cd67d89f51abbc0381 not found: ID does not exist"
Oct 30 13:07:24.390097 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.390043    4522 scope.go:122] "RemoveContainer" containerID="f3c2d524d044dbc6913cdf84b7a740c31e43c2ccab9abef1f5deefd076ef2cef"
Oct 30 13:07:24.390288 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:07:24.390271    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c2d524d044dbc6913cdf84b7a740c31e43c2ccab9abef1f5deefd076ef2cef\": container with ID starting with f3c2d524d044dbc6913cdf84b7a740c31e43c2ccab9abef1f5deefd076ef2cef not found: ID does not exist" containerID="f3c2d524d044dbc6913cdf84b7a740c31e43c2ccab9abef1f5deefd076ef2cef"
Oct 30 13:07:24.390334 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.390292    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c2d524d044dbc6913cdf84b7a740c31e43c2ccab9abef1f5deefd076ef2cef"} err="failed to get container status \"f3c2d524d044dbc6913cdf84b7a740c31e43c2ccab9abef1f5deefd076ef2cef\": rpc error: code = NotFound desc = could not find container \"f3c2d524d044dbc6913cdf84b7a740c31e43c2ccab9abef1f5deefd076ef2cef\": container with ID starting with f3c2d524d044dbc6913cdf84b7a740c31e43c2ccab9abef1f5deefd076ef2cef not found: ID does not exist"
Oct 30 13:07:24.390334 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.390306    4522 scope.go:122] "RemoveContainer" containerID="bd55715591d3d7111a83f1013c14e871c6990a14bc22f9e788a5bef9869b2c24"
Oct 30 13:07:24.398719 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.398702    4522 scope.go:122] "RemoveContainer" containerID="bd55715591d3d7111a83f1013c14e871c6990a14bc22f9e788a5bef9869b2c24"
Oct 30 13:07:24.398978 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:07:24.398958    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd55715591d3d7111a83f1013c14e871c6990a14bc22f9e788a5bef9869b2c24\": container with ID starting with bd55715591d3d7111a83f1013c14e871c6990a14bc22f9e788a5bef9869b2c24 not found: ID does not exist" containerID="bd55715591d3d7111a83f1013c14e871c6990a14bc22f9e788a5bef9869b2c24"
Oct 30 13:07:24.399036 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.398981    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd55715591d3d7111a83f1013c14e871c6990a14bc22f9e788a5bef9869b2c24"} err="failed to get container status \"bd55715591d3d7111a83f1013c14e871c6990a14bc22f9e788a5bef9869b2c24\": rpc error: code = NotFound desc = could not find container \"bd55715591d3d7111a83f1013c14e871c6990a14bc22f9e788a5bef9869b2c24\": container with ID starting with bd55715591d3d7111a83f1013c14e871c6990a14bc22f9e788a5bef9869b2c24 not found: ID does not exist"
Oct 30 13:07:24.399036 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.398995    4522 scope.go:122] "RemoveContainer" containerID="d9a9f205b0574c462664d0ee7330b56607a3d7383f1cc945538c1a7beffdae68"
Oct 30 13:07:24.406885 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.406861    4522 scope.go:122] "RemoveContainer" containerID="d9a9f205b0574c462664d0ee7330b56607a3d7383f1cc945538c1a7beffdae68"
Oct 30 13:07:24.407138 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:07:24.407120    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a9f205b0574c462664d0ee7330b56607a3d7383f1cc945538c1a7beffdae68\": container with ID starting with d9a9f205b0574c462664d0ee7330b56607a3d7383f1cc945538c1a7beffdae68 not found: ID does not exist" containerID="d9a9f205b0574c462664d0ee7330b56607a3d7383f1cc945538c1a7beffdae68"
Oct 30 13:07:24.407190 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.407145    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a9f205b0574c462664d0ee7330b56607a3d7383f1cc945538c1a7beffdae68"} err="failed to get container status \"d9a9f205b0574c462664d0ee7330b56607a3d7383f1cc945538c1a7beffdae68\": rpc error: code = NotFound desc = could not find container \"d9a9f205b0574c462664d0ee7330b56607a3d7383f1cc945538c1a7beffdae68\": container with ID starting with d9a9f205b0574c462664d0ee7330b56607a3d7383f1cc945538c1a7beffdae68 not found: ID does not exist"
Oct 30 13:07:24.407190 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.407162    4522 scope.go:122] "RemoveContainer" containerID="40f0e7b9f5e84eec9cc6d8e2030b9aecdb7f855c67aac71396e707f42c7d091b"
Oct 30 13:07:24.414772 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.414755    4522 scope.go:122] "RemoveContainer" containerID="40f0e7b9f5e84eec9cc6d8e2030b9aecdb7f855c67aac71396e707f42c7d091b"
Oct 30 13:07:24.415013 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:07:24.414996    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f0e7b9f5e84eec9cc6d8e2030b9aecdb7f855c67aac71396e707f42c7d091b\": container with ID starting with 40f0e7b9f5e84eec9cc6d8e2030b9aecdb7f855c67aac71396e707f42c7d091b not found: ID does not exist" containerID="40f0e7b9f5e84eec9cc6d8e2030b9aecdb7f855c67aac71396e707f42c7d091b"
Oct 30 13:07:24.415081 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.415016    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f0e7b9f5e84eec9cc6d8e2030b9aecdb7f855c67aac71396e707f42c7d091b"} err="failed to get container status \"40f0e7b9f5e84eec9cc6d8e2030b9aecdb7f855c67aac71396e707f42c7d091b\": rpc error: code = NotFound desc = could not find container \"40f0e7b9f5e84eec9cc6d8e2030b9aecdb7f855c67aac71396e707f42c7d091b\": container with ID starting with 40f0e7b9f5e84eec9cc6d8e2030b9aecdb7f855c67aac71396e707f42c7d091b not found: ID does not exist"
Oct 30 13:07:24.483225 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.483158    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"]
Oct 30 13:07:24.496379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.493829    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-6kmk2"]
Oct 30 13:07:24.520239 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.520191    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"]
Oct 30 13:07:24.523582 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.523557    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-j6b2v"]
Oct 30 13:07:24.559270 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.559243    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"]
Oct 30 13:07:24.568783 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.568755    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-gt9cj"]
Oct 30 13:07:24.600319 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.600259    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"]
Oct 30 13:07:24.606525 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.606505    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4cd3fb-openshift-default-79cbdb49d5-8wtm4"]
Oct 30 13:07:24.928561 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:24.928486    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:25.663002 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:25.662967    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:26.334634 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:26.334603    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" path="/var/lib/kubelet/pods/14b9e4de-979a-4e21-9714-c6a99f26d861/volumes"
Oct 30 13:07:26.335040 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:26.335026    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="674b4304-a144-449c-9c7b-bd026b669f74" path="/var/lib/kubelet/pods/674b4304-a144-449c-9c7b-bd026b669f74/volumes"
Oct 30 13:07:26.335459 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:26.335445    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" path="/var/lib/kubelet/pods/8b94e06d-f102-43e5-86f2-ab0c2065b4ca/volumes"
Oct 30 13:07:26.336147 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:26.336133    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" path="/var/lib/kubelet/pods/fa8a14d3-f3ed-43ec-aed2-8017b4f195f6/volumes"
Oct 30 13:07:28.166920 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:28.166889    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:30.475079 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:30.475044    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:30.578451 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:30.578211    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:32.334742 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:32.334704    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:34.132405 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:34.132345    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:35.814042 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:35.814007    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:37.590160 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:37.590127    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:39.354219 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:39.354187    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:41.651848 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:41.651817    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:43.808415 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:43.808381    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:45.493284 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:45.493249    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:46.107455 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:46.107418    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:47.214887 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:47.214848    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:48.904823 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:48.904791    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:49.992516 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:49.992483    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:50.181567 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:50.181531    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:50.570780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:50.570746    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 13:07:50.706278 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:50.706243    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:51.134271 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:07:51.134237    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:07:52.585385 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:52.584773    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:54.381383 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:54.380409    4522 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod674b4304-a144-449c-9c7b-bd026b669f74"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod674b4304-a144-449c-9c7b-bd026b669f74] : Timed out while waiting for systemd to remove kubepods-burstable-pod674b4304_a144_449c_9c7b_bd026b669f74.slice"
Oct 30 13:07:54.381383 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:54.380454    4522 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="674b4304-a144-449c-9c7b-bd026b669f74" err="failed to delete cgroup paths for [kubepods burstable pod674b4304-a144-449c-9c7b-bd026b669f74] : unable to destroy cgroup paths for cgroup [kubepods burstable pod674b4304-a144-449c-9c7b-bd026b669f74] : Timed out while waiting for systemd to remove kubepods-burstable-pod674b4304_a144_449c_9c7b_bd026b669f74.slice"
Oct 30 13:07:55.649112 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:55.649075    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:57.871597 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:57.871568    4522 scope.go:122] "RemoveContainer" containerID="4bb9309b2a0bbbe917d52453399acf4edbd4c3981c09bddaf397c164e7bb4916"
Oct 30 13:07:58.043793 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:58.043758    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:59.919093 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:07:59.919057    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:00.162735 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:00.162703    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hv6zx_4098fe5a-e7d3-488c-a1ed-72d447999e15/payload-processing/0.log"
Oct 30 13:08:00.357725 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:00.357699    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hv6zx_4098fe5a-e7d3-488c-a1ed-72d447999e15/payload-processing/0.log"
Oct 30 13:08:01.778155 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:01.778118    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:03.619949 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:03.619911    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:05.427762 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:05.427731    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:06.654383 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:06.654338    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"]
Oct 30 13:08:06.685328 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:06.685296    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"]
Oct 30 13:08:06.786671 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:06.786633    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:06.908569 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:06.907812    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:07.174759 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:07.174662    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:09.546851 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:09.546769    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:11.655208 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:11.655139    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx" podUID="b09c05c5-e657-4a02-bdd8-b67bd1449e78" containerName="payload-processing" containerID="cri-o://37e89a5725b295e0c922b0f7b679300e028d95b2998e9428d04cd1d08c704b3c" gracePeriod=25
Oct 30 13:08:11.685834 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:11.685803    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf" podUID="059f5d6d-e8a6-4e23-a4d9-ca0153c3638e" containerName="payload-pre-processing" containerID="cri-o://b125331ee9343588270b5dfb619b4e8d70f69091bb2ba2636c904dfc37e5259e" gracePeriod=25
Oct 30 13:08:11.908500 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:11.908431    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:13.612949 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:13.612914    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:15.409693 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:15.409662    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:15.633691 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:15.633650    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx" podUID="b09c05c5-e657-4a02-bdd8-b67bd1449e78" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:08:15.633965 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:15.633941    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf" podUID="059f5d6d-e8a6-4e23-a4d9-ca0153c3638e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:08:17.120856 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:17.120816    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:18.955045 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:18.955016    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:21.369567 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:08:21.369525    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:08:22.211312 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:22.211269    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:24.544955 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:24.544912    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:25.633773 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:25.633728    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx" podUID="b09c05c5-e657-4a02-bdd8-b67bd1449e78" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:08:25.634159 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:25.633744    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf" podUID="059f5d6d-e8a6-4e23-a4d9-ca0153c3638e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:08:26.429207 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:26.429166    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:27.902095 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:27.902063    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"]
Oct 30 13:08:27.902914 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:27.902564    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr" podUID="186f7bca-cdb1-44a8-b70d-3f02a9c891a8" containerName="maas-api" containerID="cri-o://c6f96187819a70ba40fdd928bd06de9edf7f2a14585cc95e855f90379987db07" gracePeriod=30
Oct 30 13:08:28.020542 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:28.020507    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:28.686863 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:28.686830    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:28.796389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:28.796311    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:28.868320 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:28.868281    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr" podUID="186f7bca-cdb1-44a8-b70d-3f02a9c891a8" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.127:8443/health\": dial tcp 10.133.0.127:8443: connect: connection refused"
Oct 30 13:08:29.171635 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.171611    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:08:29.239693 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.239626    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:29.273361 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.273331    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-maas-api-tls\") pod \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") "
Oct 30 13:08:29.273493 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.273382    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-metrics-certs\") pod \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") "
Oct 30 13:08:29.273493 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.273431    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-openshift-service-ca\") pod \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") "
Oct 30 13:08:29.273618 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.273506    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-kube-api-access-487z7\" (UniqueName: \"kubernetes.io/projected/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-kube-api-access-487z7\") pod \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\" (UID: \"186f7bca-cdb1-44a8-b70d-3f02a9c891a8\") "
Oct 30 13:08:29.273799 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.273775    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-openshift-service-ca" pod "186f7bca-cdb1-44a8-b70d-3f02a9c891a8" (UID: "186f7bca-cdb1-44a8-b70d-3f02a9c891a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:29.275464 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.275434    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-maas-api-tls" pod "186f7bca-cdb1-44a8-b70d-3f02a9c891a8" (UID: "186f7bca-cdb1-44a8-b70d-3f02a9c891a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:29.275573 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.275528    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-metrics-certs" pod "186f7bca-cdb1-44a8-b70d-3f02a9c891a8" (UID: "186f7bca-cdb1-44a8-b70d-3f02a9c891a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:29.275573 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.275556    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-kube-api-access-487z7" pod "186f7bca-cdb1-44a8-b70d-3f02a9c891a8" (UID: "186f7bca-cdb1-44a8-b70d-3f02a9c891a8"). InnerVolumeSpecName "kube-api-access-487z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:29.375478 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.375450    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-maas-api-tls\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:08:29.375478 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.375477    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:08:29.375698 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.375490    4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-openshift-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:08:29.375698 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.375502    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-487z7\" (UniqueName: \"kubernetes.io/projected/186f7bca-cdb1-44a8-b70d-3f02a9c891a8-kube-api-access-487z7\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:08:29.826609 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.826577    4522 generic.go:356] "Generic (PLEG): container finished" podID="186f7bca-cdb1-44a8-b70d-3f02a9c891a8" containerID="c6f96187819a70ba40fdd928bd06de9edf7f2a14585cc95e855f90379987db07" exitCode=0
Oct 30 13:08:29.826787 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.826621    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"
Oct 30 13:08:29.826787 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.826661    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr" event={"ID":"186f7bca-cdb1-44a8-b70d-3f02a9c891a8","Type":"ContainerDied","Data":"c6f96187819a70ba40fdd928bd06de9edf7f2a14585cc95e855f90379987db07"}
Oct 30 13:08:29.826787 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.826719    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr" event={"ID":"186f7bca-cdb1-44a8-b70d-3f02a9c891a8","Type":"ContainerDied","Data":"6429f217acfd40cb1e3ef856f3799023f9e5ac5b048f9686e66bb42bf23d4cc4"}
Oct 30 13:08:29.826787 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.826748    4522 scope.go:122] "RemoveContainer" containerID="c6f96187819a70ba40fdd928bd06de9edf7f2a14585cc95e855f90379987db07"
Oct 30 13:08:29.836170 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.836152    4522 scope.go:122] "RemoveContainer" containerID="c6f96187819a70ba40fdd928bd06de9edf7f2a14585cc95e855f90379987db07"
Oct 30 13:08:29.836425 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:08:29.836409    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f96187819a70ba40fdd928bd06de9edf7f2a14585cc95e855f90379987db07\": container with ID starting with c6f96187819a70ba40fdd928bd06de9edf7f2a14585cc95e855f90379987db07 not found: ID does not exist" containerID="c6f96187819a70ba40fdd928bd06de9edf7f2a14585cc95e855f90379987db07"
Oct 30 13:08:29.836491 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.836430    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f96187819a70ba40fdd928bd06de9edf7f2a14585cc95e855f90379987db07"} err="failed to get container status \"c6f96187819a70ba40fdd928bd06de9edf7f2a14585cc95e855f90379987db07\": rpc error: code = NotFound desc = could not find container \"c6f96187819a70ba40fdd928bd06de9edf7f2a14585cc95e855f90379987db07\": container with ID starting with c6f96187819a70ba40fdd928bd06de9edf7f2a14585cc95e855f90379987db07 not found: ID does not exist"
Oct 30 13:08:29.877681 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.877657    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"]
Oct 30 13:08:29.889489 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:29.889467    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cb0fe-7758dd4565-45qvr"]
Oct 30 13:08:30.334231 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:30.334196    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186f7bca-cdb1-44a8-b70d-3f02a9c891a8" path="/var/lib/kubelet/pods/186f7bca-cdb1-44a8-b70d-3f02a9c891a8/volumes"
Oct 30 13:08:30.744262 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:30.744184    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:31.858253 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:31.858220    4522 generic.go:356] "Generic (PLEG): container finished" podID="059f5d6d-e8a6-4e23-a4d9-ca0153c3638e" containerID="b125331ee9343588270b5dfb619b4e8d70f69091bb2ba2636c904dfc37e5259e" exitCode=0
Oct 30 13:08:31.858705 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:31.858287    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf" event={"ID":"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e","Type":"ContainerDied","Data":"b125331ee9343588270b5dfb619b4e8d70f69091bb2ba2636c904dfc37e5259e"}
Oct 30 13:08:31.860249 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:31.860109    4522 generic.go:356] "Generic (PLEG): container finished" podID="b09c05c5-e657-4a02-bdd8-b67bd1449e78" containerID="37e89a5725b295e0c922b0f7b679300e028d95b2998e9428d04cd1d08c704b3c" exitCode=0
Oct 30 13:08:31.860249 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:31.860185    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx" event={"ID":"b09c05c5-e657-4a02-bdd8-b67bd1449e78","Type":"ContainerDied","Data":"37e89a5725b295e0c922b0f7b679300e028d95b2998e9428d04cd1d08c704b3c"}
Oct 30 13:08:31.943517 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:31.943492    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"
Oct 30 13:08:31.983644 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:31.983621    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"
Oct 30 13:08:32.002094 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.002073    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b09c05c5-e657-4a02-bdd8-b67bd1449e78-kube-api-access-g4dcs\" (UniqueName: \"kubernetes.io/projected/b09c05c5-e657-4a02-bdd8-b67bd1449e78-kube-api-access-g4dcs\") pod \"b09c05c5-e657-4a02-bdd8-b67bd1449e78\" (UID: \"b09c05c5-e657-4a02-bdd8-b67bd1449e78\") "
Oct 30 13:08:32.002255 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.002241    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b09c05c5-e657-4a02-bdd8-b67bd1449e78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b09c05c5-e657-4a02-bdd8-b67bd1449e78-plugins-config-volume\") pod \"b09c05c5-e657-4a02-bdd8-b67bd1449e78\" (UID: \"b09c05c5-e657-4a02-bdd8-b67bd1449e78\") "
Oct 30 13:08:32.002678 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.002655    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09c05c5-e657-4a02-bdd8-b67bd1449e78-plugins-config-volume" pod "b09c05c5-e657-4a02-bdd8-b67bd1449e78" (UID: "b09c05c5-e657-4a02-bdd8-b67bd1449e78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:32.004112 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.004091    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09c05c5-e657-4a02-bdd8-b67bd1449e78-kube-api-access-g4dcs" pod "b09c05c5-e657-4a02-bdd8-b67bd1449e78" (UID: "b09c05c5-e657-4a02-bdd8-b67bd1449e78"). InnerVolumeSpecName "kube-api-access-g4dcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:32.103036 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.103013    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-plugins-config-volume\") pod \"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e\" (UID: \"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e\") "
Oct 30 13:08:32.103191 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.103077    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-kube-api-access-8ckr7\" (UniqueName: \"kubernetes.io/projected/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-kube-api-access-8ckr7\") pod \"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e\" (UID: \"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e\") "
Oct 30 13:08:32.103428 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.103401    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-plugins-config-volume" pod "059f5d6d-e8a6-4e23-a4d9-ca0153c3638e" (UID: "059f5d6d-e8a6-4e23-a4d9-ca0153c3638e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:32.103517 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.103500    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b09c05c5-e657-4a02-bdd8-b67bd1449e78-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:08:32.103517 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.103512    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:08:32.103587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.103521    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4dcs\" (UniqueName: \"kubernetes.io/projected/b09c05c5-e657-4a02-bdd8-b67bd1449e78-kube-api-access-g4dcs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:08:32.104988 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.104969    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-kube-api-access-8ckr7" pod "059f5d6d-e8a6-4e23-a4d9-ca0153c3638e" (UID: "059f5d6d-e8a6-4e23-a4d9-ca0153c3638e"). InnerVolumeSpecName "kube-api-access-8ckr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:32.204118 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.204060    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ckr7\" (UniqueName: \"kubernetes.io/projected/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e-kube-api-access-8ckr7\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:08:32.866348 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.866319    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"
Oct 30 13:08:32.866745 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.866368    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx" event={"ID":"b09c05c5-e657-4a02-bdd8-b67bd1449e78","Type":"ContainerDied","Data":"479b7cb2b7d98c2c485d50b1a0f1d00543cd5c8c3f3eaed3089217aa280d7922"}
Oct 30 13:08:32.866745 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.866439    4522 scope.go:122] "RemoveContainer" containerID="37e89a5725b295e0c922b0f7b679300e028d95b2998e9428d04cd1d08c704b3c"
Oct 30 13:08:32.867883 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.867860    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf" event={"ID":"059f5d6d-e8a6-4e23-a4d9-ca0153c3638e","Type":"ContainerDied","Data":"e87b8aaa54960157aa268e21b7af0cdd79888a97454029c73860d222c1f91f39"}
Oct 30 13:08:32.867972 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.867895    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"
Oct 30 13:08:32.875325 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.875307    4522 scope.go:122] "RemoveContainer" containerID="b125331ee9343588270b5dfb619b4e8d70f69091bb2ba2636c904dfc37e5259e"
Oct 30 13:08:32.917482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.917461    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"]
Oct 30 13:08:32.917907 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.917888    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:32.929378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.929317    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cb0fe-57656d7975-zzlzf"]
Oct 30 13:08:32.951521 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.951502    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"]
Oct 30 13:08:32.961324 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:32.961305    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cb0fe-6b4758b476-f89sx"]
Oct 30 13:08:33.108886 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:33.108862    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:34.283058 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:34.283022    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:34.333274 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:34.333239    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059f5d6d-e8a6-4e23-a4d9-ca0153c3638e" path="/var/lib/kubelet/pods/059f5d6d-e8a6-4e23-a4d9-ca0153c3638e/volumes"
Oct 30 13:08:34.333663 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:34.333648    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b09c05c5-e657-4a02-bdd8-b67bd1449e78" path="/var/lib/kubelet/pods/b09c05c5-e657-4a02-bdd8-b67bd1449e78/volumes"
Oct 30 13:08:34.905975 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:34.905945    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:36.460872 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:36.460843    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:36.646533 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:36.646504    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:36.740013 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:36.739932    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:37.058899 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:37.058843    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:37.156137 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:37.156105    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:37.286711 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:37.286659    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:37.678652 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:37.678611    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:39.291791 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:39.291750    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:41.048130 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:41.048101    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:42.739680 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:42.739650    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:43.411048 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:43.411007    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:45.220628 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:45.220588    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:45.646439 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:45.646399    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 13:08:47.013870 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:47.013802    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:49.081378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:49.080446    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:49.664992 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:49.664956    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:50.360010 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:50.359981    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:51.137910 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:08:51.137867    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:08:51.749785 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:51.749735    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:53.980565 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:53.980530    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:55.623268 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:55.623235    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:57.307294 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:57.307262    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:59.029725 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:08:59.029685    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:00.734016 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:00.733986    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:03.012942 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:03.012908    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:04.827761 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:04.827729    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:06.725226 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:06.725184    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:09.048318 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:09.048279    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:10.074313 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:10.074276    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:10.176807 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:10.176770    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:10.902010 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:10.901977    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:11.011977 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:11.011926    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:11.697251 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:11.697219    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:13.215615 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:13.215582    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:15.044538 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:15.044501    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:16.829746 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:16.829714    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:18.043477 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:18.043440    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:18.161030 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:18.160989    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:18.355594 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:18.355503    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"]
Oct 30 13:09:18.355773 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:18.355744    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy" containerID="cri-o://547c2f1fe06583e214b5623cc493e1d530d569cb5b78018a8fb1f6b33e300898" gracePeriod=30
Oct 30 13:09:18.389545 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:18.389518    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:18.692849 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:18.692755    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:20.358434 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:20.358392    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:20.622681 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:20.622598    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:20.804997 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:20.804960    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:21.133078 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:09:21.133041    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:09:22.955668 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:22.955637    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:23.618289 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.618266    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:09:23.764628 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.764559    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-credential-socket\") pod \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") "
Oct 30 13:09:23.764628 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.764616    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-envoy\") pod \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") "
Oct 30 13:09:23.764837 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.764637    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-data\") pod \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") "
Oct 30 13:09:23.764837 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.764684    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istiod-ca-cert\") pod \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") "
Oct 30 13:09:23.764837 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.764702    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-certs\") pod \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") "
Oct 30 13:09:23.764837 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.764738    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-socket\") pod \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") "
Oct 30 13:09:23.764837 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.764779    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-podinfo\") pod \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") "
Oct 30 13:09:23.764837 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.764807    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-token\" (UniqueName: \"kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-token\") pod \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") "
Oct 30 13:09:23.764837 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.764835    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-kube-api-access-twbtt\" (UniqueName: \"kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-kube-api-access-twbtt\") pod \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\" (UID: \"649cac72-df6c-4572-8fc9-a3e78a5e20b8\") "
Oct 30 13:09:23.765210 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.764846    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-credential-socket" pod "649cac72-df6c-4572-8fc9-a3e78a5e20b8" (UID: "649cac72-df6c-4572-8fc9-a3e78a5e20b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:23.765210 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.764936    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-data" pod "649cac72-df6c-4572-8fc9-a3e78a5e20b8" (UID: "649cac72-df6c-4572-8fc9-a3e78a5e20b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:23.765316 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.765246    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-socket" pod "649cac72-df6c-4572-8fc9-a3e78a5e20b8" (UID: "649cac72-df6c-4572-8fc9-a3e78a5e20b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:23.765404 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.765380    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.765482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.765403    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.765482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.765419    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.765601 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.765477    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-certs" pod "649cac72-df6c-4572-8fc9-a3e78a5e20b8" (UID: "649cac72-df6c-4572-8fc9-a3e78a5e20b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:23.765665 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.765647    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istiod-ca-cert" pod "649cac72-df6c-4572-8fc9-a3e78a5e20b8" (UID: "649cac72-df6c-4572-8fc9-a3e78a5e20b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:23.766841 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.766811    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-envoy" pod "649cac72-df6c-4572-8fc9-a3e78a5e20b8" (UID: "649cac72-df6c-4572-8fc9-a3e78a5e20b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:23.767077 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.767057    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-kube-api-access-twbtt" pod "649cac72-df6c-4572-8fc9-a3e78a5e20b8" (UID: "649cac72-df6c-4572-8fc9-a3e78a5e20b8"). InnerVolumeSpecName "kube-api-access-twbtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:23.767325 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.767297    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-podinfo" pod "649cac72-df6c-4572-8fc9-a3e78a5e20b8" (UID: "649cac72-df6c-4572-8fc9-a3e78a5e20b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:09:23.767423 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.767304    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-token" pod "649cac72-df6c-4572-8fc9-a3e78a5e20b8" (UID: "649cac72-df6c-4572-8fc9-a3e78a5e20b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:23.865934 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.865912    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.865934 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.865935    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.866081 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.865944    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/649cac72-df6c-4572-8fc9-a3e78a5e20b8-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.866081 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.865952    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.866081 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.865960    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:23.866081 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:23.865968    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twbtt\" (UniqueName: \"kubernetes.io/projected/649cac72-df6c-4572-8fc9-a3e78a5e20b8-kube-api-access-twbtt\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:24.279565 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:24.279539    4522 generic.go:356] "Generic (PLEG): container finished" podID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerID="547c2f1fe06583e214b5623cc493e1d530d569cb5b78018a8fb1f6b33e300898" exitCode=0
Oct 30 13:09:24.279962 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:24.279586    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"
Oct 30 13:09:24.279962 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:24.279601    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" event={"ID":"649cac72-df6c-4572-8fc9-a3e78a5e20b8","Type":"ContainerDied","Data":"547c2f1fe06583e214b5623cc493e1d530d569cb5b78018a8fb1f6b33e300898"}
Oct 30 13:09:24.279962 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:24.279641    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm" event={"ID":"649cac72-df6c-4572-8fc9-a3e78a5e20b8","Type":"ContainerDied","Data":"36e51498f1df149d42db91110b32f0b76e8f82172b3fcdb81a539a4511612bce"}
Oct 30 13:09:24.279962 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:24.279662    4522 scope.go:122] "RemoveContainer" containerID="547c2f1fe06583e214b5623cc493e1d530d569cb5b78018a8fb1f6b33e300898"
Oct 30 13:09:24.289344 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:24.289328    4522 scope.go:122] "RemoveContainer" containerID="a88c70190f694f652b69b8a9cd45b6c6843b3520f6166ac501897ce398e7c28c"
Oct 30 13:09:24.297092 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:24.297079    4522 scope.go:122] "RemoveContainer" containerID="547c2f1fe06583e214b5623cc493e1d530d569cb5b78018a8fb1f6b33e300898"
Oct 30 13:09:24.297382 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:09:24.297335    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547c2f1fe06583e214b5623cc493e1d530d569cb5b78018a8fb1f6b33e300898\": container with ID starting with 547c2f1fe06583e214b5623cc493e1d530d569cb5b78018a8fb1f6b33e300898 not found: ID does not exist" containerID="547c2f1fe06583e214b5623cc493e1d530d569cb5b78018a8fb1f6b33e300898"
Oct 30 13:09:24.297461 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:24.297389    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547c2f1fe06583e214b5623cc493e1d530d569cb5b78018a8fb1f6b33e300898"} err="failed to get container status \"547c2f1fe06583e214b5623cc493e1d530d569cb5b78018a8fb1f6b33e300898\": rpc error: code = NotFound desc = could not find container \"547c2f1fe06583e214b5623cc493e1d530d569cb5b78018a8fb1f6b33e300898\": container with ID starting with 547c2f1fe06583e214b5623cc493e1d530d569cb5b78018a8fb1f6b33e300898 not found: ID does not exist"
Oct 30 13:09:24.297461 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:24.297407    4522 scope.go:122] "RemoveContainer" containerID="a88c70190f694f652b69b8a9cd45b6c6843b3520f6166ac501897ce398e7c28c"
Oct 30 13:09:24.297647 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:09:24.297631    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88c70190f694f652b69b8a9cd45b6c6843b3520f6166ac501897ce398e7c28c\": container with ID starting with a88c70190f694f652b69b8a9cd45b6c6843b3520f6166ac501897ce398e7c28c not found: ID does not exist" containerID="a88c70190f694f652b69b8a9cd45b6c6843b3520f6166ac501897ce398e7c28c"
Oct 30 13:09:24.297695 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:24.297651    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88c70190f694f652b69b8a9cd45b6c6843b3520f6166ac501897ce398e7c28c"} err="failed to get container status \"a88c70190f694f652b69b8a9cd45b6c6843b3520f6166ac501897ce398e7c28c\": rpc error: code = NotFound desc = could not find container \"a88c70190f694f652b69b8a9cd45b6c6843b3520f6166ac501897ce398e7c28c\": container with ID starting with a88c70190f694f652b69b8a9cd45b6c6843b3520f6166ac501897ce398e7c28c not found: ID does not exist"
Oct 30 13:09:24.351824 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:24.351801    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"]
Oct 30 13:09:24.370736 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:24.364269    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cb0fe-openshift-default-86c9c7d99c-jdstm"]
Oct 30 13:09:24.854964 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:24.854929    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:26.350639 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:26.350577    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" path="/var/lib/kubelet/pods/649cac72-df6c-4572-8fc9-a3e78a5e20b8/volumes"
Oct 30 13:09:26.884012 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:26.883961    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:27.000390 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:27.000334    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:27.534902 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:27.527679    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:27.574377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:27.563391    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"]
Oct 30 13:09:27.574377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:27.563657    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy" containerID="cri-o://1b9e6ad45f834a16cdfb3db99c7037a011b1c837cfbb4653d0bb96be55b12cbc" gracePeriod=30
Oct 30 13:09:27.683465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:27.683420    4522 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:27.804206 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:27.803448    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq" podUID="ce95115a-fa86-4eab-9f15-1f455288c014" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:27.893816 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:27.893778    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:29.436328 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:29.436293    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"]
Oct 30 13:09:29.497634 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:29.497607    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"]
Oct 30 13:09:29.611115 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:29.611075    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:29.716723 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:29.716612    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:29.901642 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:29.901609    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:30.684954 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:30.684919    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:32.741057 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.741031    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 13:09:32.804379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.804241    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-socket\") pod \"74749d96-9482-4a69-90b3-9be8c58faa09\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") "
Oct 30 13:09:32.804379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.804300    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-istio-token\" (UniqueName: \"kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-istio-token\") pod \"74749d96-9482-4a69-90b3-9be8c58faa09\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") "
Oct 30 13:09:32.804770 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.804592    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-data\") pod \"74749d96-9482-4a69-90b3-9be8c58faa09\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") "
Oct 30 13:09:32.804770 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.804651    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-envoy\") pod \"74749d96-9482-4a69-90b3-9be8c58faa09\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") "
Oct 30 13:09:32.804770 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.804679    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74749d96-9482-4a69-90b3-9be8c58faa09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74749d96-9482-4a69-90b3-9be8c58faa09-istiod-ca-cert\") pod \"74749d96-9482-4a69-90b3-9be8c58faa09\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") "
Oct 30 13:09:32.804770 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.804706    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-credential-socket\") pod \"74749d96-9482-4a69-90b3-9be8c58faa09\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") "
Oct 30 13:09:32.805454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.805076    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74749d96-9482-4a69-90b3-9be8c58faa09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74749d96-9482-4a69-90b3-9be8c58faa09-istio-podinfo\") pod \"74749d96-9482-4a69-90b3-9be8c58faa09\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") "
Oct 30 13:09:32.805454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.805118    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-certs\") pod \"74749d96-9482-4a69-90b3-9be8c58faa09\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") "
Oct 30 13:09:32.805454 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.805226    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-kube-api-access-v988m\" (UniqueName: \"kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-kube-api-access-v988m\") pod \"74749d96-9482-4a69-90b3-9be8c58faa09\" (UID: \"74749d96-9482-4a69-90b3-9be8c58faa09\") "
Oct 30 13:09:32.807847 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.807178    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-credential-socket" pod "74749d96-9482-4a69-90b3-9be8c58faa09" (UID: "74749d96-9482-4a69-90b3-9be8c58faa09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:32.807847 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.807611    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74749d96-9482-4a69-90b3-9be8c58faa09-istiod-ca-cert" pod "74749d96-9482-4a69-90b3-9be8c58faa09" (UID: "74749d96-9482-4a69-90b3-9be8c58faa09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:32.809136 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.809112    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-socket" pod "74749d96-9482-4a69-90b3-9be8c58faa09" (UID: "74749d96-9482-4a69-90b3-9be8c58faa09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:32.812376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.809282    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-certs" pod "74749d96-9482-4a69-90b3-9be8c58faa09" (UID: "74749d96-9482-4a69-90b3-9be8c58faa09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:32.812376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.809456    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-data" pod "74749d96-9482-4a69-90b3-9be8c58faa09" (UID: "74749d96-9482-4a69-90b3-9be8c58faa09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:32.816389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.814739    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-envoy" pod "74749d96-9482-4a69-90b3-9be8c58faa09" (UID: "74749d96-9482-4a69-90b3-9be8c58faa09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:32.816389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.814841    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74749d96-9482-4a69-90b3-9be8c58faa09-istio-podinfo" pod "74749d96-9482-4a69-90b3-9be8c58faa09" (UID: "74749d96-9482-4a69-90b3-9be8c58faa09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:09:32.816389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.815194    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-istio-token" pod "74749d96-9482-4a69-90b3-9be8c58faa09" (UID: "74749d96-9482-4a69-90b3-9be8c58faa09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:32.816389 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.815516    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-kube-api-access-v988m" pod "74749d96-9482-4a69-90b3-9be8c58faa09" (UID: "74749d96-9482-4a69-90b3-9be8c58faa09"). InnerVolumeSpecName "kube-api-access-v988m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:32.907495 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.907446    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:32.907495 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.907467    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:32.907495 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.907477    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74749d96-9482-4a69-90b3-9be8c58faa09-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:32.907495 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.907486    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:32.907495 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.907494    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74749d96-9482-4a69-90b3-9be8c58faa09-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:32.907708 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.907503    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:32.907708 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.907511    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v988m\" (UniqueName: \"kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-kube-api-access-v988m\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:32.907708 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.907520    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74749d96-9482-4a69-90b3-9be8c58faa09-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:32.907708 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:32.907528    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74749d96-9482-4a69-90b3-9be8c58faa09-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:33.027853 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:33.027825    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:33.592490 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:33.592455    4522 generic.go:356] "Generic (PLEG): container finished" podID="74749d96-9482-4a69-90b3-9be8c58faa09" containerID="1b9e6ad45f834a16cdfb3db99c7037a011b1c837cfbb4653d0bb96be55b12cbc" exitCode=0
Oct 30 13:09:33.592667 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:33.592502    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"
Oct 30 13:09:33.592667 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:33.592543    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" event={"ID":"74749d96-9482-4a69-90b3-9be8c58faa09","Type":"ContainerDied","Data":"1b9e6ad45f834a16cdfb3db99c7037a011b1c837cfbb4653d0bb96be55b12cbc"}
Oct 30 13:09:33.592667 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:33.592599    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv" event={"ID":"74749d96-9482-4a69-90b3-9be8c58faa09","Type":"ContainerDied","Data":"6aeb39d18e18704017beed1ec964602974f80e2fc308d3dfb0b55e5aa61500a8"}
Oct 30 13:09:33.592667 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:33.592625    4522 scope.go:122] "RemoveContainer" containerID="1b9e6ad45f834a16cdfb3db99c7037a011b1c837cfbb4653d0bb96be55b12cbc"
Oct 30 13:09:33.602397 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:33.602380    4522 scope.go:122] "RemoveContainer" containerID="e465083f88d26f5a78107c7f001cd413bc0eb2fd7f039b92727fce371e7c8ad5"
Oct 30 13:09:33.610310 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:33.610138    4522 scope.go:122] "RemoveContainer" containerID="1b9e6ad45f834a16cdfb3db99c7037a011b1c837cfbb4653d0bb96be55b12cbc"
Oct 30 13:09:33.610419 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:09:33.610400    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9e6ad45f834a16cdfb3db99c7037a011b1c837cfbb4653d0bb96be55b12cbc\": container with ID starting with 1b9e6ad45f834a16cdfb3db99c7037a011b1c837cfbb4653d0bb96be55b12cbc not found: ID does not exist" containerID="1b9e6ad45f834a16cdfb3db99c7037a011b1c837cfbb4653d0bb96be55b12cbc"
Oct 30 13:09:33.610479 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:33.610426    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9e6ad45f834a16cdfb3db99c7037a011b1c837cfbb4653d0bb96be55b12cbc"} err="failed to get container status \"1b9e6ad45f834a16cdfb3db99c7037a011b1c837cfbb4653d0bb96be55b12cbc\": rpc error: code = NotFound desc = could not find container \"1b9e6ad45f834a16cdfb3db99c7037a011b1c837cfbb4653d0bb96be55b12cbc\": container with ID starting with 1b9e6ad45f834a16cdfb3db99c7037a011b1c837cfbb4653d0bb96be55b12cbc not found: ID does not exist"
Oct 30 13:09:33.610479 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:33.610443    4522 scope.go:122] "RemoveContainer" containerID="e465083f88d26f5a78107c7f001cd413bc0eb2fd7f039b92727fce371e7c8ad5"
Oct 30 13:09:33.610671 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:09:33.610653    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e465083f88d26f5a78107c7f001cd413bc0eb2fd7f039b92727fce371e7c8ad5\": container with ID starting with e465083f88d26f5a78107c7f001cd413bc0eb2fd7f039b92727fce371e7c8ad5 not found: ID does not exist" containerID="e465083f88d26f5a78107c7f001cd413bc0eb2fd7f039b92727fce371e7c8ad5"
Oct 30 13:09:33.610726 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:33.610682    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e465083f88d26f5a78107c7f001cd413bc0eb2fd7f039b92727fce371e7c8ad5"} err="failed to get container status \"e465083f88d26f5a78107c7f001cd413bc0eb2fd7f039b92727fce371e7c8ad5\": rpc error: code = NotFound desc = could not find container \"e465083f88d26f5a78107c7f001cd413bc0eb2fd7f039b92727fce371e7c8ad5\": container with ID starting with e465083f88d26f5a78107c7f001cd413bc0eb2fd7f039b92727fce371e7c8ad5 not found: ID does not exist"
Oct 30 13:09:33.655933 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:33.655910    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"]
Oct 30 13:09:33.661155 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:33.661135    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-026c2f-openshift-default-86445dfbc7-428zv"]
Oct 30 13:09:34.339338 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.334378    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" path="/var/lib/kubelet/pods/74749d96-9482-4a69-90b3-9be8c58faa09/volumes"
Oct 30 13:09:34.436767 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.436737    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j" podUID="d2e6e069-7ef4-4e12-b266-6f1316d1878d" containerName="payload-processing" containerID="cri-o://d86f058d2a5d33e08057396b51f8b71efe33e23fda505cf2b17c47df31dcde78" gracePeriod=25
Oct 30 13:09:34.497960 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.497935    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c" podUID="9a297ff9-2434-4474-8403-a57b1fb5f8ec" containerName="payload-pre-processing" containerID="cri-o://62680a7039803ea170879f6c8715d639c2c666fe245eccfa523c7a8e326ca3e0" gracePeriod=25
Oct 30 13:09:34.598912 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.598855    4522 generic.go:356] "Generic (PLEG): container finished" podID="d2e6e069-7ef4-4e12-b266-6f1316d1878d" containerID="d86f058d2a5d33e08057396b51f8b71efe33e23fda505cf2b17c47df31dcde78" exitCode=0
Oct 30 13:09:34.598912 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.598905    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j" event={"ID":"d2e6e069-7ef4-4e12-b266-6f1316d1878d","Type":"ContainerDied","Data":"d86f058d2a5d33e08057396b51f8b71efe33e23fda505cf2b17c47df31dcde78"}
Oct 30 13:09:34.724834 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.724246    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:34.743329 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.743307    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"
Oct 30 13:09:34.827701 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.827673    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"
Oct 30 13:09:34.829709 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.829683    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2e6e069-7ef4-4e12-b266-6f1316d1878d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e6e069-7ef4-4e12-b266-6f1316d1878d-plugins-config-volume\") pod \"d2e6e069-7ef4-4e12-b266-6f1316d1878d\" (UID: \"d2e6e069-7ef4-4e12-b266-6f1316d1878d\") "
Oct 30 13:09:34.829828 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.829743    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e6e069-7ef4-4e12-b266-6f1316d1878d-kube-api-access-w2xdp\" (UniqueName: \"kubernetes.io/projected/d2e6e069-7ef4-4e12-b266-6f1316d1878d-kube-api-access-w2xdp\") pod \"d2e6e069-7ef4-4e12-b266-6f1316d1878d\" (UID: \"d2e6e069-7ef4-4e12-b266-6f1316d1878d\") "
Oct 30 13:09:34.830039 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.830015    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e6e069-7ef4-4e12-b266-6f1316d1878d-plugins-config-volume" pod "d2e6e069-7ef4-4e12-b266-6f1316d1878d" (UID: "d2e6e069-7ef4-4e12-b266-6f1316d1878d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:34.831723 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.831703    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e6e069-7ef4-4e12-b266-6f1316d1878d-kube-api-access-w2xdp" pod "d2e6e069-7ef4-4e12-b266-6f1316d1878d" (UID: "d2e6e069-7ef4-4e12-b266-6f1316d1878d"). InnerVolumeSpecName "kube-api-access-w2xdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:34.930531 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.930477    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a297ff9-2434-4474-8403-a57b1fb5f8ec-kube-api-access-w2ts2\" (UniqueName: \"kubernetes.io/projected/9a297ff9-2434-4474-8403-a57b1fb5f8ec-kube-api-access-w2ts2\") pod \"9a297ff9-2434-4474-8403-a57b1fb5f8ec\" (UID: \"9a297ff9-2434-4474-8403-a57b1fb5f8ec\") "
Oct 30 13:09:34.930653 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.930578    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a297ff9-2434-4474-8403-a57b1fb5f8ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a297ff9-2434-4474-8403-a57b1fb5f8ec-plugins-config-volume\") pod \"9a297ff9-2434-4474-8403-a57b1fb5f8ec\" (UID: \"9a297ff9-2434-4474-8403-a57b1fb5f8ec\") "
Oct 30 13:09:34.930864 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.930844    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e6e069-7ef4-4e12-b266-6f1316d1878d-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:34.930917 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.930871    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2xdp\" (UniqueName: \"kubernetes.io/projected/d2e6e069-7ef4-4e12-b266-6f1316d1878d-kube-api-access-w2xdp\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:34.930962 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.930949    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a297ff9-2434-4474-8403-a57b1fb5f8ec-plugins-config-volume" pod "9a297ff9-2434-4474-8403-a57b1fb5f8ec" (UID: "9a297ff9-2434-4474-8403-a57b1fb5f8ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:34.932396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:34.932374    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a297ff9-2434-4474-8403-a57b1fb5f8ec-kube-api-access-w2ts2" pod "9a297ff9-2434-4474-8403-a57b1fb5f8ec" (UID: "9a297ff9-2434-4474-8403-a57b1fb5f8ec"). InnerVolumeSpecName "kube-api-access-w2ts2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:35.032170 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.032143    4522 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a297ff9-2434-4474-8403-a57b1fb5f8ec-plugins-config-volume\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:35.032170 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.032166    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2ts2\" (UniqueName: \"kubernetes.io/projected/9a297ff9-2434-4474-8403-a57b1fb5f8ec-kube-api-access-w2ts2\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:35.605174 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.605149    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"
Oct 30 13:09:35.605639 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.605184    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j" event={"ID":"d2e6e069-7ef4-4e12-b266-6f1316d1878d","Type":"ContainerDied","Data":"9e95e1beadea3f355884dcdb0558e8161b4785ef4c8e340af31cab372fec068c"}
Oct 30 13:09:35.605639 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.605247    4522 scope.go:122] "RemoveContainer" containerID="d86f058d2a5d33e08057396b51f8b71efe33e23fda505cf2b17c47df31dcde78"
Oct 30 13:09:35.606628 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.606600    4522 generic.go:356] "Generic (PLEG): container finished" podID="9a297ff9-2434-4474-8403-a57b1fb5f8ec" containerID="62680a7039803ea170879f6c8715d639c2c666fe245eccfa523c7a8e326ca3e0" exitCode=0
Oct 30 13:09:35.606738 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.606632    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c" event={"ID":"9a297ff9-2434-4474-8403-a57b1fb5f8ec","Type":"ContainerDied","Data":"62680a7039803ea170879f6c8715d639c2c666fe245eccfa523c7a8e326ca3e0"}
Oct 30 13:09:35.606738 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.606663    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c" event={"ID":"9a297ff9-2434-4474-8403-a57b1fb5f8ec","Type":"ContainerDied","Data":"30c53226c79b173b5d2e418625186fa8b68622824afa64e510454ccac938a063"}
Oct 30 13:09:35.606738 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.606663    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"
Oct 30 13:09:35.614973 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.614786    4522 scope.go:122] "RemoveContainer" containerID="62680a7039803ea170879f6c8715d639c2c666fe245eccfa523c7a8e326ca3e0"
Oct 30 13:09:35.623414 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.623391    4522 scope.go:122] "RemoveContainer" containerID="62680a7039803ea170879f6c8715d639c2c666fe245eccfa523c7a8e326ca3e0"
Oct 30 13:09:35.623670 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:09:35.623647    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62680a7039803ea170879f6c8715d639c2c666fe245eccfa523c7a8e326ca3e0\": container with ID starting with 62680a7039803ea170879f6c8715d639c2c666fe245eccfa523c7a8e326ca3e0 not found: ID does not exist" containerID="62680a7039803ea170879f6c8715d639c2c666fe245eccfa523c7a8e326ca3e0"
Oct 30 13:09:35.623729 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.623678    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62680a7039803ea170879f6c8715d639c2c666fe245eccfa523c7a8e326ca3e0"} err="failed to get container status \"62680a7039803ea170879f6c8715d639c2c666fe245eccfa523c7a8e326ca3e0\": rpc error: code = NotFound desc = could not find container \"62680a7039803ea170879f6c8715d639c2c666fe245eccfa523c7a8e326ca3e0\": container with ID starting with 62680a7039803ea170879f6c8715d639c2c666fe245eccfa523c7a8e326ca3e0 not found: ID does not exist"
Oct 30 13:09:35.660923 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.660898    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"]
Oct 30 13:09:35.672332 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.672310    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8f3cb8-8d7576f7f-wct4c"]
Oct 30 13:09:35.699984 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.699961    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"]
Oct 30 13:09:35.706996 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:35.706950    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8f3cb8-5b659b4974-wpv8j"]
Oct 30 13:09:36.334936 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:36.334905    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a297ff9-2434-4474-8403-a57b1fb5f8ec" path="/var/lib/kubelet/pods/9a297ff9-2434-4474-8403-a57b1fb5f8ec/volumes"
Oct 30 13:09:36.335265 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:36.335252    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2e6e069-7ef4-4e12-b266-6f1316d1878d" path="/var/lib/kubelet/pods/d2e6e069-7ef4-4e12-b266-6f1316d1878d/volumes"
Oct 30 13:09:36.516587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:36.516559    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:38.337391 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:38.337337    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:38.458696 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:38.458667    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:38.571072 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:38.571000    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:39.083509 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:39.083473    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:39.183747 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:39.183711    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:39.281695 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:39.281664    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:39.383953 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:39.383875    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 13:09:39.769884 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:39.769854    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:41.258073 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:41.258028    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:43.796329 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:43.796293    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:45.438087 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:45.438044    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:45.539130 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:45.539096    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:45.750241 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:45.750150    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"]
Oct 30 13:09:45.750467 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:45.750435    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerName="istio-proxy" containerID="cri-o://b7baac47ed14a6de54425781dab6471a975832ad3f31d1696147905df164e6f8" gracePeriod=30
Oct 30 13:09:45.795655 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:45.795619    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:46.582289 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:46.582256    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:46.931234 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:46.931141    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"]
Oct 30 13:09:46.931485 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:46.931451    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy" containerID="cri-o://b69004c6c828972b1726a0e68680172c81bb34690b53669ca4d60816feffeb6d" gracePeriod=30
Oct 30 13:09:47.478676 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:47.478639    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:49.323501 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:49.323411    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:50.209946 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.209917    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 13:09:50.351069 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351040    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"]
Oct 30 13:09:50.351468 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351453    4522 cpu_manager.go:414] "containerMap: removing container" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy"
Oct 30 13:09:50.351539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351470    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy"
Oct 30 13:09:50.351539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351481    4522 cpu_manager.go:414] "containerMap: removing container" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main"
Oct 30 13:09:50.351539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351486    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main"
Oct 30 13:09:50.351539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351498    4522 cpu_manager.go:414] "containerMap: removing container" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy"
Oct 30 13:09:50.351539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351503    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy"
Oct 30 13:09:50.351539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351510    4522 cpu_manager.go:414] "containerMap: removing container" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy"
Oct 30 13:09:50.351539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351515    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy"
Oct 30 13:09:50.351539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351522    4522 cpu_manager.go:414] "containerMap: removing container" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerName="istio-proxy"
Oct 30 13:09:50.351539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351527    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerName="istio-proxy"
Oct 30 13:09:50.351539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351533    4522 cpu_manager.go:414] "containerMap: removing container" podUID="b09c05c5-e657-4a02-bdd8-b67bd1449e78" containerName="payload-processing"
Oct 30 13:09:50.351539 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351537    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b09c05c5-e657-4a02-bdd8-b67bd1449e78" containerName="payload-processing"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351546    4522 cpu_manager.go:414] "containerMap: removing container" podUID="9a297ff9-2434-4474-8403-a57b1fb5f8ec" containerName="payload-pre-processing"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351551    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a297ff9-2434-4474-8403-a57b1fb5f8ec" containerName="payload-pre-processing"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351561    4522 cpu_manager.go:414] "containerMap: removing container" podUID="186f7bca-cdb1-44a8-b70d-3f02a9c891a8" containerName="maas-api"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351566    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186f7bca-cdb1-44a8-b70d-3f02a9c891a8" containerName="maas-api"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351572    4522 cpu_manager.go:414] "containerMap: removing container" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351577    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351583    4522 cpu_manager.go:414] "containerMap: removing container" podUID="f382c2bb-0885-48f3-9ac1-09585a2b1c36" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351588    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f382c2bb-0885-48f3-9ac1-09585a2b1c36" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351595    4522 cpu_manager.go:414] "containerMap: removing container" podUID="b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878" containerName="payload-processing"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351600    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878" containerName="payload-processing"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351608    4522 cpu_manager.go:414] "containerMap: removing container" podUID="059f5d6d-e8a6-4e23-a4d9-ca0153c3638e" containerName="payload-pre-processing"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351612    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059f5d6d-e8a6-4e23-a4d9-ca0153c3638e" containerName="payload-pre-processing"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351620    4522 cpu_manager.go:414] "containerMap: removing container" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351625    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351634    4522 cpu_manager.go:414] "containerMap: removing container" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351638    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351646    4522 cpu_manager.go:414] "containerMap: removing container" podUID="753b333f-cc81-4df9-ad51-e14445ce8fa4" containerName="maas-api"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351650    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="753b333f-cc81-4df9-ad51-e14445ce8fa4" containerName="maas-api"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351655    4522 cpu_manager.go:414] "containerMap: removing container" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351659    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351665    4522 cpu_manager.go:414] "containerMap: removing container" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351669    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351676    4522 cpu_manager.go:414] "containerMap: removing container" podUID="b68996c3-3081-47c8-9878-d2e438a298a7" containerName="discovery"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351680    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68996c3-3081-47c8-9878-d2e438a298a7" containerName="discovery"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351687    4522 cpu_manager.go:414] "containerMap: removing container" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351691    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351698    4522 cpu_manager.go:414] "containerMap: removing container" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351702    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351707    4522 cpu_manager.go:414] "containerMap: removing container" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351711    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351717    4522 cpu_manager.go:414] "containerMap: removing container" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351722    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351727    4522 cpu_manager.go:414] "containerMap: removing container" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351732    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351741    4522 cpu_manager.go:414] "containerMap: removing container" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351746    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351751    4522 cpu_manager.go:414] "containerMap: removing container" podUID="186f443b-4af4-40dd-ad37-71be0c18ebf4" containerName="maas-api"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351756    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186f443b-4af4-40dd-ad37-71be0c18ebf4" containerName="maas-api"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351762    4522 cpu_manager.go:414] "containerMap: removing container" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351767    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351774    4522 cpu_manager.go:414] "containerMap: removing container" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351780    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351788    4522 cpu_manager.go:414] "containerMap: removing container" podUID="d2e6e069-7ef4-4e12-b266-6f1316d1878d" containerName="payload-processing"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351792    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e6e069-7ef4-4e12-b266-6f1316d1878d" containerName="payload-processing"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351797    4522 cpu_manager.go:414] "containerMap: removing container" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351802    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351812    4522 cpu_manager.go:414] "containerMap: removing container" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351816    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351823    4522 cpu_manager.go:414] "containerMap: removing container" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351828    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351836    4522 cpu_manager.go:414] "containerMap: removing container" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351840    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351846    4522 cpu_manager.go:414] "containerMap: removing container" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351850    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351910    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="753b333f-cc81-4df9-ad51-e14445ce8fa4" containerName="maas-api"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351918    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2019baf-7b8e-4c05-96b8-b48f93809985" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351924    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfeffa41-ac7c-4809-a0c0-fa9b2e9412ef" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351930    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68996c3-3081-47c8-9878-d2e438a298a7" containerName="discovery"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351936    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351941    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1616ab-319c-4608-893b-fdcc65631488" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351947    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351953    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351957    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ef2bf6-d815-459d-8162-2dc297e01897" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351963    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6aa45d0-5cee-463c-a2d4-89ee83f50aa1" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351968    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="186f443b-4af4-40dd-ad37-71be0c18ebf4" containerName="maas-api"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351972    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59f4327-f9c2-4445-97b6-63a99043d625" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351978    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351986    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b94e06d-f102-43e5-86f2-ab0c2065b4ca" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.351996    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8a14d3-f3ed-43ec-aed2-8017b4f195f6" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352004    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e6e069-7ef4-4e12-b266-6f1316d1878d" containerName="payload-processing"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352012    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3fda1a0-5841-4eaa-9621-1fe8b5165b12" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352020    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="849bf94f-d57d-4f16-8494-09bbbf8e3db9" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352027    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352033    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45b4a4c-94e4-4897-8523-0e446e994bea" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352038    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e6f2bb2-7f4c-4672-a201-9b822bea3687" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352044    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="2423ab7f-0469-463d-9aaf-b17895f4c4a6" containerName="main"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352049    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352055    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="e726d2a8-8f05-471d-a7d1-2fa2c8df6550" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352060    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="208f0cbd-3ed2-4e90-9929-277352e907d3" containerName="istio-proxy"
Oct 30 13:09:50.352020 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352065    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b09c05c5-e657-4a02-bdd8-b67bd1449e78" containerName="payload-processing"
Oct 30 13:09:50.354481 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352071    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a297ff9-2434-4474-8403-a57b1fb5f8ec" containerName="payload-pre-processing"
Oct 30 13:09:50.354481 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352077    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy"
Oct 30 13:09:50.354481 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352082    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="186f7bca-cdb1-44a8-b70d-3f02a9c891a8" containerName="maas-api"
Oct 30 13:09:50.354481 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352088    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f382c2bb-0885-48f3-9ac1-09585a2b1c36" containerName="istio-proxy"
Oct 30 13:09:50.354481 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352093    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="b530d6b3-7fb9-4e37-ba74-c9a1bb0d3878" containerName="payload-processing"
Oct 30 13:09:50.354481 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352098    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="059f5d6d-e8a6-4e23-a4d9-ca0153c3638e" containerName="payload-pre-processing"
Oct 30 13:09:50.354481 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352102    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b9e4de-979a-4e21-9714-c6a99f26d861" containerName="istio-proxy"
Oct 30 13:09:50.354481 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.352107    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="03285671-c2a3-4503-accb-e6e3b41a16a5" containerName="main"
Oct 30 13:09:50.356396 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.356378    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"
Oct 30 13:09:50.372524 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.372502    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"]
Oct 30 13:09:50.381755 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.381732    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:09:50.381847 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.381778    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:09:50.452014 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.451987    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q55d6\" (UniqueName: \"kubernetes.io/projected/165a54eb-a4c8-4193-9d01-952415abb48f-kube-api-access-q55d6\") pod \"maas-api-revoke-sub-0315ecbb-n48gd\" (UID: \"165a54eb-a4c8-4193-9d01-952415abb48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"
Oct 30 13:09:50.452133 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.452065    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0315ecbb-n48gd\" (UID: \"165a54eb-a4c8-4193-9d01-952415abb48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"
Oct 30 13:09:50.452181 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.452164    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0315ecbb-n48gd\" (UID: \"165a54eb-a4c8-4193-9d01-952415abb48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"
Oct 30 13:09:50.499376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.499289    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:50.552875 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.552850    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0315ecbb-n48gd\" (UID: \"165a54eb-a4c8-4193-9d01-952415abb48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"
Oct 30 13:09:50.552971 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.552890    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0315ecbb-n48gd\" (UID: \"165a54eb-a4c8-4193-9d01-952415abb48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"
Oct 30 13:09:50.552971 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.552935    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q55d6\" (UniqueName: \"kubernetes.io/projected/165a54eb-a4c8-4193-9d01-952415abb48f-kube-api-access-q55d6\") pod \"maas-api-revoke-sub-0315ecbb-n48gd\" (UID: \"165a54eb-a4c8-4193-9d01-952415abb48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"
Oct 30 13:09:50.553446 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.553429    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0315ecbb-n48gd\" (UID: \"165a54eb-a4c8-4193-9d01-952415abb48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"
Oct 30 13:09:50.555086 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.555070    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0315ecbb-n48gd\" (UID: \"165a54eb-a4c8-4193-9d01-952415abb48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"
Oct 30 13:09:50.571520 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.571493    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q55d6\" (UniqueName: \"kubernetes.io/projected/165a54eb-a4c8-4193-9d01-952415abb48f-kube-api-access-q55d6\") pod \"maas-api-revoke-sub-0315ecbb-n48gd\" (UID: \"165a54eb-a4c8-4193-9d01-952415abb48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"
Oct 30 13:09:50.669176 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.669156    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"
Oct 30 13:09:50.971063 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:09:50.971030    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165a54eb_a4c8_4193_9d01_952415abb48f.slice/crio-43eab7f02dd8a520e39707f349177fa2fc4a57c03437adf2a903a708d2e10f2a WatchSource:0}: Error finding container 43eab7f02dd8a520e39707f349177fa2fc4a57c03437adf2a903a708d2e10f2a: Status 404 returned error can't find the container with id 43eab7f02dd8a520e39707f349177fa2fc4a57c03437adf2a903a708d2e10f2a
Oct 30 13:09:50.971375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.971326    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"]
Oct 30 13:09:50.994834 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:50.994816    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:09:51.027726 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.027702    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:51.060186 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.060159    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-envoy\") pod \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") "
Oct 30 13:09:51.060289 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.060219    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-socket\") pod \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") "
Oct 30 13:09:51.060289 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.060274    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-podinfo\") pod \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") "
Oct 30 13:09:51.060404 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.060324    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-token\" (UniqueName: \"kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-token\") pod \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") "
Oct 30 13:09:51.060514 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.060480    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istiod-ca-cert\") pod \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") "
Oct 30 13:09:51.060627 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.060559    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-data\") pod \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") "
Oct 30 13:09:51.060627 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.060488    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-socket" pod "cb69b573-fa0e-4789-80e7-8df8caef1cb7" (UID: "cb69b573-fa0e-4789-80e7-8df8caef1cb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:51.060764 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.060654    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-certs\") pod \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") "
Oct 30 13:09:51.060764 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.060698    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-credential-socket\") pod \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") "
Oct 30 13:09:51.060880 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.060759    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-kube-api-access-dtgbn\" (UniqueName: \"kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-kube-api-access-dtgbn\") pod \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\" (UID: \"cb69b573-fa0e-4789-80e7-8df8caef1cb7\") "
Oct 30 13:09:51.060880 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.060828    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-data" pod "cb69b573-fa0e-4789-80e7-8df8caef1cb7" (UID: "cb69b573-fa0e-4789-80e7-8df8caef1cb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:51.060880 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.060776    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istiod-ca-cert" pod "cb69b573-fa0e-4789-80e7-8df8caef1cb7" (UID: "cb69b573-fa0e-4789-80e7-8df8caef1cb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:51.061040 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.061003    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-credential-socket" pod "cb69b573-fa0e-4789-80e7-8df8caef1cb7" (UID: "cb69b573-fa0e-4789-80e7-8df8caef1cb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:51.061040 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.061016    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-certs" pod "cb69b573-fa0e-4789-80e7-8df8caef1cb7" (UID: "cb69b573-fa0e-4789-80e7-8df8caef1cb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:51.061179 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.061160    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:51.061280 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.061179    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:51.061280 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.061196    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:51.061280 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.061209    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:51.061280 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.061221    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:51.062697 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.062657    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-envoy" pod "cb69b573-fa0e-4789-80e7-8df8caef1cb7" (UID: "cb69b573-fa0e-4789-80e7-8df8caef1cb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:51.062825 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.062801    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-podinfo" pod "cb69b573-fa0e-4789-80e7-8df8caef1cb7" (UID: "cb69b573-fa0e-4789-80e7-8df8caef1cb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:09:51.062988 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.062967    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-token" pod "cb69b573-fa0e-4789-80e7-8df8caef1cb7" (UID: "cb69b573-fa0e-4789-80e7-8df8caef1cb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:51.063071 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.062996    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-kube-api-access-dtgbn" pod "cb69b573-fa0e-4789-80e7-8df8caef1cb7" (UID: "cb69b573-fa0e-4789-80e7-8df8caef1cb7"). InnerVolumeSpecName "kube-api-access-dtgbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:51.133171 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:09:51.133136    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:09:51.162991 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.162875    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtgbn\" (UniqueName: \"kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-kube-api-access-dtgbn\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:51.162991 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.162921    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:51.162991 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.162954    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:51.162991 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.162966    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb69b573-fa0e-4789-80e7-8df8caef1cb7-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:51.728622 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.728528    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd" event={"ID":"165a54eb-a4c8-4193-9d01-952415abb48f","Type":"ContainerStarted","Data":"c9eb04259625398754b320bf7eb072282f4fb69498d8a5c8db009130918195f1"}
Oct 30 13:09:51.728622 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.728601    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd" event={"ID":"165a54eb-a4c8-4193-9d01-952415abb48f","Type":"ContainerStarted","Data":"43eab7f02dd8a520e39707f349177fa2fc4a57c03437adf2a903a708d2e10f2a"}
Oct 30 13:09:51.730867 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.730838    4522 generic.go:356] "Generic (PLEG): container finished" podID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerID="b7baac47ed14a6de54425781dab6471a975832ad3f31d1696147905df164e6f8" exitCode=0
Oct 30 13:09:51.731004 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.730893    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"
Oct 30 13:09:51.731004 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.730918    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" event={"ID":"cb69b573-fa0e-4789-80e7-8df8caef1cb7","Type":"ContainerDied","Data":"b7baac47ed14a6de54425781dab6471a975832ad3f31d1696147905df164e6f8"}
Oct 30 13:09:51.731004 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.730969    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv" event={"ID":"cb69b573-fa0e-4789-80e7-8df8caef1cb7","Type":"ContainerDied","Data":"5943893e5891fbe1a3e72c1a6825b9a0115b0d28e9d281c7c684652600e56e60"}
Oct 30 13:09:51.731004 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.730998    4522 scope.go:122] "RemoveContainer" containerID="b7baac47ed14a6de54425781dab6471a975832ad3f31d1696147905df164e6f8"
Oct 30 13:09:51.746670 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.746648    4522 scope.go:122] "RemoveContainer" containerID="78065e469c98a58f34a7f18f8c4890205c0af25924b3ae184c8b93a90631063a"
Oct 30 13:09:51.756591 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.756567    4522 scope.go:122] "RemoveContainer" containerID="b7baac47ed14a6de54425781dab6471a975832ad3f31d1696147905df164e6f8"
Oct 30 13:09:51.756861 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:09:51.756841    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7baac47ed14a6de54425781dab6471a975832ad3f31d1696147905df164e6f8\": container with ID starting with b7baac47ed14a6de54425781dab6471a975832ad3f31d1696147905df164e6f8 not found: ID does not exist" containerID="b7baac47ed14a6de54425781dab6471a975832ad3f31d1696147905df164e6f8"
Oct 30 13:09:51.756956 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.756872    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7baac47ed14a6de54425781dab6471a975832ad3f31d1696147905df164e6f8"} err="failed to get container status \"b7baac47ed14a6de54425781dab6471a975832ad3f31d1696147905df164e6f8\": rpc error: code = NotFound desc = could not find container \"b7baac47ed14a6de54425781dab6471a975832ad3f31d1696147905df164e6f8\": container with ID starting with b7baac47ed14a6de54425781dab6471a975832ad3f31d1696147905df164e6f8 not found: ID does not exist"
Oct 30 13:09:51.756956 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.756894    4522 scope.go:122] "RemoveContainer" containerID="78065e469c98a58f34a7f18f8c4890205c0af25924b3ae184c8b93a90631063a"
Oct 30 13:09:51.757224 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:09:51.757197    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78065e469c98a58f34a7f18f8c4890205c0af25924b3ae184c8b93a90631063a\": container with ID starting with 78065e469c98a58f34a7f18f8c4890205c0af25924b3ae184c8b93a90631063a not found: ID does not exist" containerID="78065e469c98a58f34a7f18f8c4890205c0af25924b3ae184c8b93a90631063a"
Oct 30 13:09:51.757338 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.757229    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78065e469c98a58f34a7f18f8c4890205c0af25924b3ae184c8b93a90631063a"} err="failed to get container status \"78065e469c98a58f34a7f18f8c4890205c0af25924b3ae184c8b93a90631063a\": rpc error: code = NotFound desc = could not find container \"78065e469c98a58f34a7f18f8c4890205c0af25924b3ae184c8b93a90631063a\": container with ID starting with 78065e469c98a58f34a7f18f8c4890205c0af25924b3ae184c8b93a90631063a not found: ID does not exist"
Oct 30 13:09:51.771262 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.771201    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd" podStartSLOduration=1.771185043 podStartE2EDuration="1.771185043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:51.769737575 +0000 UTC m=+2035.585064069" watchObservedRunningTime="2026-10-30 13:09:51.771185043 +0000 UTC m=+2035.586511534"
Oct 30 13:09:51.807331 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.807284    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"]
Oct 30 13:09:51.818723 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:51.818697    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-8f3cb8-openshift-default-95c47cbb9-qprhv"]
Oct 30 13:09:52.575376 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.574446    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" path="/var/lib/kubelet/pods/cb69b573-fa0e-4789-80e7-8df8caef1cb7/volumes"
Oct 30 13:09:52.749944 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.749918    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:09:52.824673 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.824640    4522 generic.go:356] "Generic (PLEG): container finished" podID="165a54eb-a4c8-4193-9d01-952415abb48f" containerID="c9eb04259625398754b320bf7eb072282f4fb69498d8a5c8db009130918195f1" exitCode=0
Oct 30 13:09:52.824938 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.824914    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd" event={"ID":"165a54eb-a4c8-4193-9d01-952415abb48f","Type":"ContainerDied","Data":"c9eb04259625398754b320bf7eb072282f4fb69498d8a5c8db009130918195f1"}
Oct 30 13:09:52.827573 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.827510    4522 generic.go:356] "Generic (PLEG): container finished" podID="1300d77d-620e-4675-8673-23ac3742083a" containerID="b69004c6c828972b1726a0e68680172c81bb34690b53669ca4d60816feffeb6d" exitCode=0
Oct 30 13:09:52.827965 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.827849    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"
Oct 30 13:09:52.828189 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.827862    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" event={"ID":"1300d77d-620e-4675-8673-23ac3742083a","Type":"ContainerDied","Data":"b69004c6c828972b1726a0e68680172c81bb34690b53669ca4d60816feffeb6d"}
Oct 30 13:09:52.828316 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.828301    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp" event={"ID":"1300d77d-620e-4675-8673-23ac3742083a","Type":"ContainerDied","Data":"f89880eefcb5fbbfeaad6896afe8f497acf243153b422e4c1269a5e9aebc2b41"}
Oct 30 13:09:52.861377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.857390    4522 scope.go:122] "RemoveContainer" containerID="b69004c6c828972b1726a0e68680172c81bb34690b53669ca4d60816feffeb6d"
Oct 30 13:09:52.921377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.919736    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-kube-api-access-pjb7r\" (UniqueName: \"kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-kube-api-access-pjb7r\") pod \"1300d77d-620e-4675-8673-23ac3742083a\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") "
Oct 30 13:09:52.921377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.919820    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-envoy\") pod \"1300d77d-620e-4675-8673-23ac3742083a\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") "
Oct 30 13:09:52.921377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.919871    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-data\") pod \"1300d77d-620e-4675-8673-23ac3742083a\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") "
Oct 30 13:09:52.921377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.919922    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1300d77d-620e-4675-8673-23ac3742083a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1300d77d-620e-4675-8673-23ac3742083a-istiod-ca-cert\") pod \"1300d77d-620e-4675-8673-23ac3742083a\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") "
Oct 30 13:09:52.921377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.920006    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-istio-token\" (UniqueName: \"kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-istio-token\") pod \"1300d77d-620e-4675-8673-23ac3742083a\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") "
Oct 30 13:09:52.921377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.920037    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-socket\") pod \"1300d77d-620e-4675-8673-23ac3742083a\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") "
Oct 30 13:09:52.921377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.920062    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-certs\") pod \"1300d77d-620e-4675-8673-23ac3742083a\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") "
Oct 30 13:09:52.921377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.920105    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1300d77d-620e-4675-8673-23ac3742083a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1300d77d-620e-4675-8673-23ac3742083a-istio-podinfo\") pod \"1300d77d-620e-4675-8673-23ac3742083a\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") "
Oct 30 13:09:52.921377 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.920148    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-credential-socket\") pod \"1300d77d-620e-4675-8673-23ac3742083a\" (UID: \"1300d77d-620e-4675-8673-23ac3742083a\") "
Oct 30 13:09:52.960375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:52.945471    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-credential-socket" pod "1300d77d-620e-4675-8673-23ac3742083a" (UID: "1300d77d-620e-4675-8673-23ac3742083a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:53.032671 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.011305    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-data" pod "1300d77d-620e-4675-8673-23ac3742083a" (UID: "1300d77d-620e-4675-8673-23ac3742083a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:53.032671 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.011450    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-certs" pod "1300d77d-620e-4675-8673-23ac3742083a" (UID: "1300d77d-620e-4675-8673-23ac3742083a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:53.032671 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.011773    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1300d77d-620e-4675-8673-23ac3742083a-istiod-ca-cert" pod "1300d77d-620e-4675-8673-23ac3742083a" (UID: "1300d77d-620e-4675-8673-23ac3742083a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:53.032671 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.021798    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:53.032671 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.021825    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:53.032671 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.021840    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1300d77d-620e-4675-8673-23ac3742083a-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:53.032671 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.021855    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:53.032671 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.022678    4522 scope.go:122] "RemoveContainer" containerID="a5d7baab680b3f004bd8c41d310fad53bc9259084dac28a00525fbaeaf0ce064"
Oct 30 13:09:53.032671 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.025736    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-kube-api-access-pjb7r" pod "1300d77d-620e-4675-8673-23ac3742083a" (UID: "1300d77d-620e-4675-8673-23ac3742083a"). InnerVolumeSpecName "kube-api-access-pjb7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:53.036797 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.036755    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-socket" pod "1300d77d-620e-4675-8673-23ac3742083a" (UID: "1300d77d-620e-4675-8673-23ac3742083a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:53.093195 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.070487    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1300d77d-620e-4675-8673-23ac3742083a-istio-podinfo" pod "1300d77d-620e-4675-8673-23ac3742083a" (UID: "1300d77d-620e-4675-8673-23ac3742083a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:09:53.119622 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.119520    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-istio-token" pod "1300d77d-620e-4675-8673-23ac3742083a" (UID: "1300d77d-620e-4675-8673-23ac3742083a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:53.121296 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.119850    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-envoy" pod "1300d77d-620e-4675-8673-23ac3742083a" (UID: "1300d77d-620e-4675-8673-23ac3742083a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:53.122615 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.122591    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjb7r\" (UniqueName: \"kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-kube-api-access-pjb7r\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:53.122779 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.122763    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:53.122881 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.122868    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1300d77d-620e-4675-8673-23ac3742083a-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:53.123004 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.122989    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1300d77d-620e-4675-8673-23ac3742083a-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:53.123101 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.123088    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1300d77d-620e-4675-8673-23ac3742083a-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:53.152209 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.151884    4522 scope.go:122] "RemoveContainer" containerID="b69004c6c828972b1726a0e68680172c81bb34690b53669ca4d60816feffeb6d"
Oct 30 13:09:53.153082 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:09:53.152762    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69004c6c828972b1726a0e68680172c81bb34690b53669ca4d60816feffeb6d\": container with ID starting with b69004c6c828972b1726a0e68680172c81bb34690b53669ca4d60816feffeb6d not found: ID does not exist" containerID="b69004c6c828972b1726a0e68680172c81bb34690b53669ca4d60816feffeb6d"
Oct 30 13:09:53.153082 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.152798    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69004c6c828972b1726a0e68680172c81bb34690b53669ca4d60816feffeb6d"} err="failed to get container status \"b69004c6c828972b1726a0e68680172c81bb34690b53669ca4d60816feffeb6d\": rpc error: code = NotFound desc = could not find container \"b69004c6c828972b1726a0e68680172c81bb34690b53669ca4d60816feffeb6d\": container with ID starting with b69004c6c828972b1726a0e68680172c81bb34690b53669ca4d60816feffeb6d not found: ID does not exist"
Oct 30 13:09:53.153082 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.152827    4522 scope.go:122] "RemoveContainer" containerID="a5d7baab680b3f004bd8c41d310fad53bc9259084dac28a00525fbaeaf0ce064"
Oct 30 13:09:53.153536 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:09:53.153445    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d7baab680b3f004bd8c41d310fad53bc9259084dac28a00525fbaeaf0ce064\": container with ID starting with a5d7baab680b3f004bd8c41d310fad53bc9259084dac28a00525fbaeaf0ce064 not found: ID does not exist" containerID="a5d7baab680b3f004bd8c41d310fad53bc9259084dac28a00525fbaeaf0ce064"
Oct 30 13:09:53.153536 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.153477    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d7baab680b3f004bd8c41d310fad53bc9259084dac28a00525fbaeaf0ce064"} err="failed to get container status \"a5d7baab680b3f004bd8c41d310fad53bc9259084dac28a00525fbaeaf0ce064\": rpc error: code = NotFound desc = could not find container \"a5d7baab680b3f004bd8c41d310fad53bc9259084dac28a00525fbaeaf0ce064\": container with ID starting with a5d7baab680b3f004bd8c41d310fad53bc9259084dac28a00525fbaeaf0ce064 not found: ID does not exist"
Oct 30 13:09:53.312378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.312153    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"]
Oct 30 13:09:53.333378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.329694    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:53.349533 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:53.349386    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a81a48-openshift-default-5fdddf48dc-hjnlp"]
Oct 30 13:09:54.243422 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.240502    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"
Oct 30 13:09:54.346333 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.346296    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-openshift-ca\") pod \"165a54eb-a4c8-4193-9d01-952415abb48f\" (UID: \"165a54eb-a4c8-4193-9d01-952415abb48f\") "
Oct 30 13:09:54.346746 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.346719    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-openshift-ca" pod "165a54eb-a4c8-4193-9d01-952415abb48f" (UID: "165a54eb-a4c8-4193-9d01-952415abb48f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:54.346850 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.346721    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-xks-ca\") pod \"165a54eb-a4c8-4193-9d01-952415abb48f\" (UID: \"165a54eb-a4c8-4193-9d01-952415abb48f\") "
Oct 30 13:09:54.346850 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.346797    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165a54eb-a4c8-4193-9d01-952415abb48f-kube-api-access-q55d6\" (UniqueName: \"kubernetes.io/projected/165a54eb-a4c8-4193-9d01-952415abb48f-kube-api-access-q55d6\") pod \"165a54eb-a4c8-4193-9d01-952415abb48f\" (UID: \"165a54eb-a4c8-4193-9d01-952415abb48f\") "
Oct 30 13:09:54.347066 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.347051    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-openshift-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:54.350000 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.349751    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165a54eb-a4c8-4193-9d01-952415abb48f-kube-api-access-q55d6" pod "165a54eb-a4c8-4193-9d01-952415abb48f" (UID: "165a54eb-a4c8-4193-9d01-952415abb48f"). InnerVolumeSpecName "kube-api-access-q55d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:54.351311 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.351285    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-xks-ca" pod "165a54eb-a4c8-4193-9d01-952415abb48f" (UID: "165a54eb-a4c8-4193-9d01-952415abb48f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:54.351465 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.351444    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1300d77d-620e-4675-8673-23ac3742083a" path="/var/lib/kubelet/pods/1300d77d-620e-4675-8673-23ac3742083a/volumes"
Oct 30 13:09:54.448510 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.448454    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/165a54eb-a4c8-4193-9d01-952415abb48f-maas-api-xks-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:54.448510 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.448485    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q55d6\" (UniqueName: \"kubernetes.io/projected/165a54eb-a4c8-4193-9d01-952415abb48f-kube-api-access-q55d6\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:09:54.848992 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.848953    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd" event={"ID":"165a54eb-a4c8-4193-9d01-952415abb48f","Type":"ContainerDied","Data":"43eab7f02dd8a520e39707f349177fa2fc4a57c03437adf2a903a708d2e10f2a"}
Oct 30 13:09:54.848992 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.848997    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0315ecbb-n48gd"
Oct 30 13:09:54.849199 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:54.849013    4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43eab7f02dd8a520e39707f349177fa2fc4a57c03437adf2a903a708d2e10f2a"
Oct 30 13:09:55.528964 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:55.528934    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:56.721286 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:56.721250    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 13:09:57.309031 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:57.308999    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:59.815375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:09:59.815278    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:00.940959 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:00.940925    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:01.490061 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.490028    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:01.724231 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724198    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"]
Oct 30 13:10:01.724601 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724587    4522 cpu_manager.go:414] "containerMap: removing container" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerName="istio-proxy"
Oct 30 13:10:01.724662 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724602    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerName="istio-proxy"
Oct 30 13:10:01.724662 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724625    4522 cpu_manager.go:414] "containerMap: removing container" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy"
Oct 30 13:10:01.724662 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724630    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy"
Oct 30 13:10:01.724662 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724642    4522 cpu_manager.go:414] "containerMap: removing container" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy"
Oct 30 13:10:01.724662 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724647    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e76d3f0-c1f4-4458-be8d-8d6922001217" containerName="istio-proxy"
Oct 30 13:10:01.724662 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724658    4522 cpu_manager.go:414] "containerMap: removing container" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy"
Oct 30 13:10:01.724662 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724662    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy"
Oct 30 13:10:01.724932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724669    4522 cpu_manager.go:414] "containerMap: removing container" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy"
Oct 30 13:10:01.724932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724673    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy"
Oct 30 13:10:01.724932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724683    4522 cpu_manager.go:414] "containerMap: removing container" podUID="165a54eb-a4c8-4193-9d01-952415abb48f" containerName="revoke-keys"
Oct 30 13:10:01.724932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724690    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165a54eb-a4c8-4193-9d01-952415abb48f" containerName="revoke-keys"
Oct 30 13:10:01.724932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724706    4522 cpu_manager.go:414] "containerMap: removing container" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy"
Oct 30 13:10:01.724932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724710    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="649cac72-df6c-4572-8fc9-a3e78a5e20b8" containerName="istio-proxy"
Oct 30 13:10:01.724932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724775    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerName="istio-proxy"
Oct 30 13:10:01.724932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724784    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="674b4304-a144-449c-9c7b-bd026b669f74" containerName="istio-proxy"
Oct 30 13:10:01.724932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724789    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerName="istio-proxy"
Oct 30 13:10:01.724932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724795    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="74749d96-9482-4a69-90b3-9be8c58faa09" containerName="istio-proxy"
Oct 30 13:10:01.724932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724802    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy"
Oct 30 13:10:01.724932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724809    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="1300d77d-620e-4675-8673-23ac3742083a" containerName="istio-proxy"
Oct 30 13:10:01.724932 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.724815    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="165a54eb-a4c8-4193-9d01-952415abb48f" containerName="revoke-keys"
Oct 30 13:10:01.729380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.729343    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"
Oct 30 13:10:01.744074 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.744015    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:10:01.746483 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.746461    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"]
Oct 30 13:10:01.746577 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.746467    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:10:01.810147 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.810121    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n\" (UID: \"233bf778-f71c-4b7e-b395-04b76f0a5bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"
Oct 30 13:10:01.810277 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.810189    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n77h\" (UniqueName: \"kubernetes.io/projected/233bf778-f71c-4b7e-b395-04b76f0a5bde-kube-api-access-9n77h\") pod \"maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n\" (UID: \"233bf778-f71c-4b7e-b395-04b76f0a5bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"
Oct 30 13:10:01.810341 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.810288    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n\" (UID: \"233bf778-f71c-4b7e-b395-04b76f0a5bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"
Oct 30 13:10:01.911070 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.911043    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n\" (UID: \"233bf778-f71c-4b7e-b395-04b76f0a5bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"
Oct 30 13:10:01.911185 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.911114    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n77h\" (UniqueName: \"kubernetes.io/projected/233bf778-f71c-4b7e-b395-04b76f0a5bde-kube-api-access-9n77h\") pod \"maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n\" (UID: \"233bf778-f71c-4b7e-b395-04b76f0a5bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"
Oct 30 13:10:01.911185 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.911137    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n\" (UID: \"233bf778-f71c-4b7e-b395-04b76f0a5bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"
Oct 30 13:10:01.911714 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.911689    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n\" (UID: \"233bf778-f71c-4b7e-b395-04b76f0a5bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"
Oct 30 13:10:01.913313 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.913294    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n\" (UID: \"233bf778-f71c-4b7e-b395-04b76f0a5bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"
Oct 30 13:10:01.934563 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:01.934538    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n77h\" (UniqueName: \"kubernetes.io/projected/233bf778-f71c-4b7e-b395-04b76f0a5bde-kube-api-access-9n77h\") pod \"maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n\" (UID: \"233bf778-f71c-4b7e-b395-04b76f0a5bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"
Oct 30 13:10:02.042596 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:02.042568    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"
Oct 30 13:10:02.347955 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:10:02.347925    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233bf778_f71c_4b7e_b395_04b76f0a5bde.slice/crio-97bb43a7216f4b6c09211a978e5a34103d5e6cd67363c0f9394e996755b6114d WatchSource:0}: Error finding container 97bb43a7216f4b6c09211a978e5a34103d5e6cd67363c0f9394e996755b6114d: Status 404 returned error can't find the container with id 97bb43a7216f4b6c09211a978e5a34103d5e6cd67363c0f9394e996755b6114d
Oct 30 13:10:02.348059 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:02.347983    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"]
Oct 30 13:10:02.459325 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:02.459162    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:02.958326 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:02.958260    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n" event={"ID":"233bf778-f71c-4b7e-b395-04b76f0a5bde","Type":"ContainerStarted","Data":"a0ad843a1eb791d72b5ce10f721c0416a81f7bb8c8b6918269cd227a22f5679a"}
Oct 30 13:10:02.958470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:02.958339    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n" event={"ID":"233bf778-f71c-4b7e-b395-04b76f0a5bde","Type":"ContainerStarted","Data":"97bb43a7216f4b6c09211a978e5a34103d5e6cd67363c0f9394e996755b6114d"}
Oct 30 13:10:02.998012 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:02.997970    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n" podStartSLOduration=1.9979593439999999 podStartE2EDuration="1.997959344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:02.996613171 +0000 UTC m=+2046.811939660" watchObservedRunningTime="2026-10-30 13:10:02.997959344 +0000 UTC m=+2046.813285831"
Oct 30 13:10:03.964712 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:03.964679    4522 generic.go:356] "Generic (PLEG): container finished" podID="233bf778-f71c-4b7e-b395-04b76f0a5bde" containerID="a0ad843a1eb791d72b5ce10f721c0416a81f7bb8c8b6918269cd227a22f5679a" exitCode=0
Oct 30 13:10:03.965100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:03.964724    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n" event={"ID":"233bf778-f71c-4b7e-b395-04b76f0a5bde","Type":"ContainerDied","Data":"a0ad843a1eb791d72b5ce10f721c0416a81f7bb8c8b6918269cd227a22f5679a"}
Oct 30 13:10:04.117930 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:04.117896    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:05.107804 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.107773    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"
Oct 30 13:10:05.159486 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.159457    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 13:10:05.241611 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.241539    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-openshift-ca\") pod \"233bf778-f71c-4b7e-b395-04b76f0a5bde\" (UID: \"233bf778-f71c-4b7e-b395-04b76f0a5bde\") "
Oct 30 13:10:05.241611 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.241593    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-xks-ca\") pod \"233bf778-f71c-4b7e-b395-04b76f0a5bde\" (UID: \"233bf778-f71c-4b7e-b395-04b76f0a5bde\") "
Oct 30 13:10:05.241797 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.241674    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233bf778-f71c-4b7e-b395-04b76f0a5bde-kube-api-access-9n77h\" (UniqueName: \"kubernetes.io/projected/233bf778-f71c-4b7e-b395-04b76f0a5bde-kube-api-access-9n77h\") pod \"233bf778-f71c-4b7e-b395-04b76f0a5bde\" (UID: \"233bf778-f71c-4b7e-b395-04b76f0a5bde\") "
Oct 30 13:10:05.241946 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.241922    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-openshift-ca" pod "233bf778-f71c-4b7e-b395-04b76f0a5bde" (UID: "233bf778-f71c-4b7e-b395-04b76f0a5bde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:05.242050 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.242035    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-openshift-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:05.243575 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.243553    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-xks-ca" pod "233bf778-f71c-4b7e-b395-04b76f0a5bde" (UID: "233bf778-f71c-4b7e-b395-04b76f0a5bde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:05.243936 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.243913    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233bf778-f71c-4b7e-b395-04b76f0a5bde-kube-api-access-9n77h" pod "233bf778-f71c-4b7e-b395-04b76f0a5bde" (UID: "233bf778-f71c-4b7e-b395-04b76f0a5bde"). InnerVolumeSpecName "kube-api-access-9n77h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:05.343449 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.343418    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n77h\" (UniqueName: \"kubernetes.io/projected/233bf778-f71c-4b7e-b395-04b76f0a5bde-kube-api-access-9n77h\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:05.343449 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.343445    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/233bf778-f71c-4b7e-b395-04b76f0a5bde-maas-api-xks-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:05.918443 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.918407    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:05.976115 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.976085    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"
Oct 30 13:10:05.976286 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.976085    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n" event={"ID":"233bf778-f71c-4b7e-b395-04b76f0a5bde","Type":"ContainerDied","Data":"97bb43a7216f4b6c09211a978e5a34103d5e6cd67363c0f9394e996755b6114d"}
Oct 30 13:10:05.976286 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:05.976251    4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97bb43a7216f4b6c09211a978e5a34103d5e6cd67363c0f9394e996755b6114d"
Oct 30 13:10:06.029391 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:06.029349    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:06.463153 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:06.463121    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"]
Oct 30 13:10:06.490882 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:06.490853    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"]
Oct 30 13:10:06.493075 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:06.493050    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"]
Oct 30 13:10:06.639853 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:06.639822    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:08.821187 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:08.821140    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:10.585558 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:10.585527    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:11.757223 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:11.757189    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:11.862206 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:11.862159    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:12.343527 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:12.343476    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:12.465902 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:12.465839    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:12.592999 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:12.592962    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:13.853383 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.853341    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"]
Oct 30 13:10:13.853736 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.853719    4522 cpu_manager.go:414] "containerMap: removing container" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerName="istio-proxy"
Oct 30 13:10:13.853736 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.853730    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb69b573-fa0e-4789-80e7-8df8caef1cb7" containerName="istio-proxy"
Oct 30 13:10:13.853829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.853745    4522 cpu_manager.go:414] "containerMap: removing container" podUID="233bf778-f71c-4b7e-b395-04b76f0a5bde" containerName="revoke-keys"
Oct 30 13:10:13.853829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.853751    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233bf778-f71c-4b7e-b395-04b76f0a5bde" containerName="revoke-keys"
Oct 30 13:10:13.853829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.853822    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="233bf778-f71c-4b7e-b395-04b76f0a5bde" containerName="revoke-keys"
Oct 30 13:10:13.857181 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.857161    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"
Oct 30 13:10:13.877378 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.877327    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"]
Oct 30 13:10:13.903388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.903340    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3375bbdf-rn848\" (UID: \"a8b12332-aec4-4639-8acb-c7534329da3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"
Oct 30 13:10:13.903494 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.903414    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3375bbdf-rn848\" (UID: \"a8b12332-aec4-4639-8acb-c7534329da3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"
Oct 30 13:10:13.903554 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.903536    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ff9j\" (UniqueName: \"kubernetes.io/projected/a8b12332-aec4-4639-8acb-c7534329da3b-kube-api-access-2ff9j\") pod \"maas-api-revoke-sub-3375bbdf-rn848\" (UID: \"a8b12332-aec4-4639-8acb-c7534329da3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"
Oct 30 13:10:13.950122 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.950097    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"]
Oct 30 13:10:13.953802 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.953785    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"
Oct 30 13:10:13.955596 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.955574    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:10:13.955842 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.955825    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pfwpx\""
Oct 30 13:10:13.971032 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:13.971013    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"]
Oct 30 13:10:14.004747 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.004721    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ff9j\" (UniqueName: \"kubernetes.io/projected/a8b12332-aec4-4639-8acb-c7534329da3b-kube-api-access-2ff9j\") pod \"maas-api-revoke-sub-3375bbdf-rn848\" (UID: \"a8b12332-aec4-4639-8acb-c7534329da3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"
Oct 30 13:10:14.004856 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.004780    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9d9a42-dcvjg\" (UID: \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"
Oct 30 13:10:14.004856 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.004845    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3375bbdf-rn848\" (UID: \"a8b12332-aec4-4639-8acb-c7534329da3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"
Oct 30 13:10:14.004955 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.004895    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3375bbdf-rn848\" (UID: \"a8b12332-aec4-4639-8acb-c7534329da3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"
Oct 30 13:10:14.004955 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.004917    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9d9a42-dcvjg\" (UID: \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"
Oct 30 13:10:14.004955 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.004943    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7bd\" (UniqueName: \"kubernetes.io/projected/db0791c0-56d1-427d-a1a5-2ec1f1af434b-kube-api-access-gv7bd\") pod \"maas-api-revoke-sub-0f9d9a42-dcvjg\" (UID: \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"
Oct 30 13:10:14.005561 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.005538    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3375bbdf-rn848\" (UID: \"a8b12332-aec4-4639-8acb-c7534329da3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"
Oct 30 13:10:14.007089 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.007074    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3375bbdf-rn848\" (UID: \"a8b12332-aec4-4639-8acb-c7534329da3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"
Oct 30 13:10:14.024095 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.024073    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ff9j\" (UniqueName: \"kubernetes.io/projected/a8b12332-aec4-4639-8acb-c7534329da3b-kube-api-access-2ff9j\") pod \"maas-api-revoke-sub-3375bbdf-rn848\" (UID: \"a8b12332-aec4-4639-8acb-c7534329da3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"
Oct 30 13:10:14.105936 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.105881    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9d9a42-dcvjg\" (UID: \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"
Oct 30 13:10:14.105936 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.105923    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9d9a42-dcvjg\" (UID: \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"
Oct 30 13:10:14.106079 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.105940    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7bd\" (UniqueName: \"kubernetes.io/projected/db0791c0-56d1-427d-a1a5-2ec1f1af434b-kube-api-access-gv7bd\") pod \"maas-api-revoke-sub-0f9d9a42-dcvjg\" (UID: \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"
Oct 30 13:10:14.106510 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.106488    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9d9a42-dcvjg\" (UID: \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"
Oct 30 13:10:14.107956 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.107934    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9d9a42-dcvjg\" (UID: \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"
Oct 30 13:10:14.127788 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.127209    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7bd\" (UniqueName: \"kubernetes.io/projected/db0791c0-56d1-427d-a1a5-2ec1f1af434b-kube-api-access-gv7bd\") pod \"maas-api-revoke-sub-0f9d9a42-dcvjg\" (UID: \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"
Oct 30 13:10:14.173722 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.173701    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"
Oct 30 13:10:14.266038 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.266009    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"
Oct 30 13:10:14.471564 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.471535    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"]
Oct 30 13:10:14.472114 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:10:14.472083    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b12332_aec4_4639_8acb_c7534329da3b.slice/crio-9936201e5ea708232125d0b3147a448a76d1d03228624f1e48773bb3a8c14019 WatchSource:0}: Error finding container 9936201e5ea708232125d0b3147a448a76d1d03228624f1e48773bb3a8c14019: Status 404 returned error can't find the container with id 9936201e5ea708232125d0b3147a448a76d1d03228624f1e48773bb3a8c14019
Oct 30 13:10:14.563181 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:14.563156    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"]
Oct 30 13:10:14.563301 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:10:14.563277    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0791c0_56d1_427d_a1a5_2ec1f1af434b.slice/crio-c85c8f90b7ae7998e979b6d6003991686ce6fb0ad6152830e30bda12d4fd18a8 WatchSource:0}: Error finding container c85c8f90b7ae7998e979b6d6003991686ce6fb0ad6152830e30bda12d4fd18a8: Status 404 returned error can't find the container with id c85c8f90b7ae7998e979b6d6003991686ce6fb0ad6152830e30bda12d4fd18a8
Oct 30 13:10:15.046236 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:15.046134    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848" event={"ID":"a8b12332-aec4-4639-8acb-c7534329da3b","Type":"ContainerStarted","Data":"220783112e6de216e609897f04324ea07d96cd5bca430e6746d4bedd111937cb"}
Oct 30 13:10:15.046236 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:15.046193    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848" event={"ID":"a8b12332-aec4-4639-8acb-c7534329da3b","Type":"ContainerStarted","Data":"9936201e5ea708232125d0b3147a448a76d1d03228624f1e48773bb3a8c14019"}
Oct 30 13:10:15.047686 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:15.047655    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg" event={"ID":"db0791c0-56d1-427d-a1a5-2ec1f1af434b","Type":"ContainerStarted","Data":"cef72c34c2635159a3fa1c7e29020dff15b638ce9b0129e75db13a3dd61a1e6d"}
Oct 30 13:10:15.047782 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:15.047704    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg" event={"ID":"db0791c0-56d1-427d-a1a5-2ec1f1af434b","Type":"ContainerStarted","Data":"c85c8f90b7ae7998e979b6d6003991686ce6fb0ad6152830e30bda12d4fd18a8"}
Oct 30 13:10:15.085575 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:15.085515    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848" podStartSLOduration=2.085496932 podStartE2EDuration="2.085496932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:15.084672716 +0000 UTC m=+2058.899999205" watchObservedRunningTime="2026-10-30 13:10:15.085496932 +0000 UTC m=+2058.900823428"
Oct 30 13:10:15.127057 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:15.127005    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg" podStartSLOduration=2.126992504 podStartE2EDuration="2.126992504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:15.124119122 +0000 UTC m=+2058.939445611" watchObservedRunningTime="2026-10-30 13:10:15.126992504 +0000 UTC m=+2058.942318993"
Oct 30 13:10:16.055198 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:16.055164    4522 generic.go:356] "Generic (PLEG): container finished" podID="a8b12332-aec4-4639-8acb-c7534329da3b" containerID="220783112e6de216e609897f04324ea07d96cd5bca430e6746d4bedd111937cb" exitCode=0
Oct 30 13:10:16.055639 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:16.055257    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848" event={"ID":"a8b12332-aec4-4639-8acb-c7534329da3b","Type":"ContainerDied","Data":"220783112e6de216e609897f04324ea07d96cd5bca430e6746d4bedd111937cb"}
Oct 30 13:10:16.056685 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:16.056664    4522 generic.go:356] "Generic (PLEG): container finished" podID="db0791c0-56d1-427d-a1a5-2ec1f1af434b" containerID="cef72c34c2635159a3fa1c7e29020dff15b638ce9b0129e75db13a3dd61a1e6d" exitCode=0
Oct 30 13:10:16.056803 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:16.056715    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg" event={"ID":"db0791c0-56d1-427d-a1a5-2ec1f1af434b","Type":"ContainerDied","Data":"cef72c34c2635159a3fa1c7e29020dff15b638ce9b0129e75db13a3dd61a1e6d"}
Oct 30 13:10:17.207031 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.207000    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"
Oct 30 13:10:17.236804 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.236769    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-xks-ca\") pod \"a8b12332-aec4-4639-8acb-c7534329da3b\" (UID: \"a8b12332-aec4-4639-8acb-c7534329da3b\") "
Oct 30 13:10:17.236950 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.236832    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b12332-aec4-4639-8acb-c7534329da3b-kube-api-access-2ff9j\" (UniqueName: \"kubernetes.io/projected/a8b12332-aec4-4639-8acb-c7534329da3b-kube-api-access-2ff9j\") pod \"a8b12332-aec4-4639-8acb-c7534329da3b\" (UID: \"a8b12332-aec4-4639-8acb-c7534329da3b\") "
Oct 30 13:10:17.236950 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.236903    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-openshift-ca\") pod \"a8b12332-aec4-4639-8acb-c7534329da3b\" (UID: \"a8b12332-aec4-4639-8acb-c7534329da3b\") "
Oct 30 13:10:17.237304 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.237272    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-openshift-ca" pod "a8b12332-aec4-4639-8acb-c7534329da3b" (UID: "a8b12332-aec4-4639-8acb-c7534329da3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:17.238931 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.238893    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-xks-ca" pod "a8b12332-aec4-4639-8acb-c7534329da3b" (UID: "a8b12332-aec4-4639-8acb-c7534329da3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:17.239035 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.238981    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b12332-aec4-4639-8acb-c7534329da3b-kube-api-access-2ff9j" pod "a8b12332-aec4-4639-8acb-c7534329da3b" (UID: "a8b12332-aec4-4639-8acb-c7534329da3b"). InnerVolumeSpecName "kube-api-access-2ff9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:17.243716 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.243694    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"
Oct 30 13:10:17.337767 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.337684    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-xks-ca\") pod \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\" (UID: \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\") "
Oct 30 13:10:17.337767 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.337722    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0791c0-56d1-427d-a1a5-2ec1f1af434b-kube-api-access-gv7bd\" (UniqueName: \"kubernetes.io/projected/db0791c0-56d1-427d-a1a5-2ec1f1af434b-kube-api-access-gv7bd\") pod \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\" (UID: \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\") "
Oct 30 13:10:17.337969 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.337808    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-openshift-ca\") pod \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\" (UID: \"db0791c0-56d1-427d-a1a5-2ec1f1af434b\") "
Oct 30 13:10:17.338034 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.338021    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-xks-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.338076 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.338037    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ff9j\" (UniqueName: \"kubernetes.io/projected/a8b12332-aec4-4639-8acb-c7534329da3b-kube-api-access-2ff9j\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.338076 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.338047    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b12332-aec4-4639-8acb-c7534329da3b-maas-api-openshift-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.338201 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.338178    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-openshift-ca" pod "db0791c0-56d1-427d-a1a5-2ec1f1af434b" (UID: "db0791c0-56d1-427d-a1a5-2ec1f1af434b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:17.339835 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.339812    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0791c0-56d1-427d-a1a5-2ec1f1af434b-kube-api-access-gv7bd" pod "db0791c0-56d1-427d-a1a5-2ec1f1af434b" (UID: "db0791c0-56d1-427d-a1a5-2ec1f1af434b"). InnerVolumeSpecName "kube-api-access-gv7bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:17.339915 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.339855    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-xks-ca" pod "db0791c0-56d1-427d-a1a5-2ec1f1af434b" (UID: "db0791c0-56d1-427d-a1a5-2ec1f1af434b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:17.438882 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.438845    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-xks-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.438882 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.438873    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv7bd\" (UniqueName: \"kubernetes.io/projected/db0791c0-56d1-427d-a1a5-2ec1f1af434b-kube-api-access-gv7bd\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.438882 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:17.438883    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0791c0-56d1-427d-a1a5-2ec1f1af434b-maas-api-openshift-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.068927 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:18.068889    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848"
Oct 30 13:10:18.069191 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:18.068892    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375bbdf-rn848" event={"ID":"a8b12332-aec4-4639-8acb-c7534329da3b","Type":"ContainerDied","Data":"9936201e5ea708232125d0b3147a448a76d1d03228624f1e48773bb3a8c14019"}
Oct 30 13:10:18.069191 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:18.069031    4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9936201e5ea708232125d0b3147a448a76d1d03228624f1e48773bb3a8c14019"
Oct 30 13:10:18.070307 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:18.070283    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg"
Oct 30 13:10:18.070479 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:18.070308    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9d9a42-dcvjg" event={"ID":"db0791c0-56d1-427d-a1a5-2ec1f1af434b","Type":"ContainerDied","Data":"c85c8f90b7ae7998e979b6d6003991686ce6fb0ad6152830e30bda12d4fd18a8"}
Oct 30 13:10:18.070479 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:18.070374    4522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c85c8f90b7ae7998e979b6d6003991686ce6fb0ad6152830e30bda12d4fd18a8"
Oct 30 13:10:18.376380 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:18.376286    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:18.478731 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:18.478696    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:21.132990 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:10:21.132956    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:10:21.482714 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.482615    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main" containerID="cri-o://fd6262173ff2c2ddab9746adf517d6dbc7a7ce879d203c8e760827673abd7409" gracePeriod=45
Oct 30 13:10:21.510289 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.510251    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main" containerID="cri-o://2be2447f716cab56852b14f7e3e5e3882c73fdf42ffc53caa75b3fda08dfa752" gracePeriod=45
Oct 30 13:10:21.513704 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.513664    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main" containerID="cri-o://5f081f824c8cac562cfe1f79d493132dca3d84b12983a511986e28b363f8a861" gracePeriod=45
Oct 30 13:10:21.895372 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.892805    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:10:21.932262 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.932231    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:10:21.934780 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.934755    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:10:21.985211 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985158    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f886013f-7c33-4731-84e8-4dcff7f18fd3-tls-certs\" (UniqueName: \"kubernetes.io/secret/f886013f-7c33-4731-84e8-4dcff7f18fd3-tls-certs\") pod \"f886013f-7c33-4731-84e8-4dcff7f18fd3\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") "
Oct 30 13:10:21.985453 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985227    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-home\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-home\") pod \"76b974f0-872d-457a-b442-71e652a012cc\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") "
Oct 30 13:10:21.985453 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985271    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76b974f0-872d-457a-b442-71e652a012cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/76b974f0-872d-457a-b442-71e652a012cc-tls-certs\") pod \"76b974f0-872d-457a-b442-71e652a012cc\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") "
Oct 30 13:10:21.985453 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985302    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-model-cache\") pod \"f886013f-7c33-4731-84e8-4dcff7f18fd3\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") "
Oct 30 13:10:21.985453 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985333    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-dshm\") pod \"76b974f0-872d-457a-b442-71e652a012cc\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") "
Oct 30 13:10:21.985453 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985421    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-tmp-dir\") pod \"29c562c6-1c99-4494-9436-137188543b90\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") "
Oct 30 13:10:21.985829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985486    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b974f0-872d-457a-b442-71e652a012cc-kube-api-access-pvb89\" (UniqueName: \"kubernetes.io/projected/76b974f0-872d-457a-b442-71e652a012cc-kube-api-access-pvb89\") pod \"76b974f0-872d-457a-b442-71e652a012cc\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") "
Oct 30 13:10:21.985829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985564    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-dshm\") pod \"f886013f-7c33-4731-84e8-4dcff7f18fd3\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") "
Oct 30 13:10:21.985829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985599    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-model-cache\") pod \"29c562c6-1c99-4494-9436-137188543b90\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") "
Oct 30 13:10:21.985829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985629    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-tmp-dir\") pod \"76b974f0-872d-457a-b442-71e652a012cc\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") "
Oct 30 13:10:21.985829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985655    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f886013f-7c33-4731-84e8-4dcff7f18fd3-kube-api-access-26fpf\" (UniqueName: \"kubernetes.io/projected/f886013f-7c33-4731-84e8-4dcff7f18fd3-kube-api-access-26fpf\") pod \"f886013f-7c33-4731-84e8-4dcff7f18fd3\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") "
Oct 30 13:10:21.985829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985686    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-tmp-dir\") pod \"f886013f-7c33-4731-84e8-4dcff7f18fd3\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") "
Oct 30 13:10:21.985829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985713    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c562c6-1c99-4494-9436-137188543b90-kube-api-access-gq2pv\" (UniqueName: \"kubernetes.io/projected/29c562c6-1c99-4494-9436-137188543b90-kube-api-access-gq2pv\") pod \"29c562c6-1c99-4494-9436-137188543b90\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") "
Oct 30 13:10:21.985829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985750    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-home\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-home\") pod \"29c562c6-1c99-4494-9436-137188543b90\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") "
Oct 30 13:10:21.985829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985779    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-dshm\") pod \"29c562c6-1c99-4494-9436-137188543b90\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") "
Oct 30 13:10:21.985829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985816    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-model-cache\") pod \"76b974f0-872d-457a-b442-71e652a012cc\" (UID: \"76b974f0-872d-457a-b442-71e652a012cc\") "
Oct 30 13:10:21.986406 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985857    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29c562c6-1c99-4494-9436-137188543b90-tls-certs\" (UniqueName: \"kubernetes.io/secret/29c562c6-1c99-4494-9436-137188543b90-tls-certs\") pod \"29c562c6-1c99-4494-9436-137188543b90\" (UID: \"29c562c6-1c99-4494-9436-137188543b90\") "
Oct 30 13:10:21.986406 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.985902    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-home\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-home\") pod \"f886013f-7c33-4731-84e8-4dcff7f18fd3\" (UID: \"f886013f-7c33-4731-84e8-4dcff7f18fd3\") "
Oct 30 13:10:21.986590 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.986561    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-home" pod "f886013f-7c33-4731-84e8-4dcff7f18fd3" (UID: "f886013f-7c33-4731-84e8-4dcff7f18fd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:21.986802 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.986778    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-tmp-dir" pod "76b974f0-872d-457a-b442-71e652a012cc" (UID: "76b974f0-872d-457a-b442-71e652a012cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:21.987032 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.987006    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-model-cache" pod "29c562c6-1c99-4494-9436-137188543b90" (UID: "29c562c6-1c99-4494-9436-137188543b90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:21.988084 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.987837    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-model-cache" pod "76b974f0-872d-457a-b442-71e652a012cc" (UID: "76b974f0-872d-457a-b442-71e652a012cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:21.988840 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.988814    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-tmp-dir" pod "f886013f-7c33-4731-84e8-4dcff7f18fd3" (UID: "f886013f-7c33-4731-84e8-4dcff7f18fd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:21.989261 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.989241    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-home" pod "29c562c6-1c99-4494-9436-137188543b90" (UID: "29c562c6-1c99-4494-9436-137188543b90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:21.989636 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.989617    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-home" pod "76b974f0-872d-457a-b442-71e652a012cc" (UID: "76b974f0-872d-457a-b442-71e652a012cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:21.993426 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.993307    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f886013f-7c33-4731-84e8-4dcff7f18fd3-kube-api-access-26fpf" pod "f886013f-7c33-4731-84e8-4dcff7f18fd3" (UID: "f886013f-7c33-4731-84e8-4dcff7f18fd3"). InnerVolumeSpecName "kube-api-access-26fpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:22.003267 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.996711    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b974f0-872d-457a-b442-71e652a012cc-kube-api-access-pvb89" pod "76b974f0-872d-457a-b442-71e652a012cc" (UID: "76b974f0-872d-457a-b442-71e652a012cc"). InnerVolumeSpecName "kube-api-access-pvb89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:22.003267 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.996863    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-tmp-dir" pod "29c562c6-1c99-4494-9436-137188543b90" (UID: "29c562c6-1c99-4494-9436-137188543b90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:22.003267 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:21.996988    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-model-cache" pod "f886013f-7c33-4731-84e8-4dcff7f18fd3" (UID: "f886013f-7c33-4731-84e8-4dcff7f18fd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:22.003267 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.001067    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b974f0-872d-457a-b442-71e652a012cc-tls-certs" pod "76b974f0-872d-457a-b442-71e652a012cc" (UID: "76b974f0-872d-457a-b442-71e652a012cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:22.003267 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.002291    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-dshm" pod "29c562c6-1c99-4494-9436-137188543b90" (UID: "29c562c6-1c99-4494-9436-137188543b90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:22.003267 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.002393    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c562c6-1c99-4494-9436-137188543b90-tls-certs" pod "29c562c6-1c99-4494-9436-137188543b90" (UID: "29c562c6-1c99-4494-9436-137188543b90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:22.003267 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.002456    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f886013f-7c33-4731-84e8-4dcff7f18fd3-tls-certs" pod "f886013f-7c33-4731-84e8-4dcff7f18fd3" (UID: "f886013f-7c33-4731-84e8-4dcff7f18fd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:22.003267 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.002791    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c562c6-1c99-4494-9436-137188543b90-kube-api-access-gq2pv" pod "29c562c6-1c99-4494-9436-137188543b90" (UID: "29c562c6-1c99-4494-9436-137188543b90"). InnerVolumeSpecName "kube-api-access-gq2pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:22.003267 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.003087    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-dshm" pod "76b974f0-872d-457a-b442-71e652a012cc" (UID: "76b974f0-872d-457a-b442-71e652a012cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:22.006132 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.004454    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-dshm" pod "f886013f-7c33-4731-84e8-4dcff7f18fd3" (UID: "f886013f-7c33-4731-84e8-4dcff7f18fd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.087838    4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-dshm\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.087875    4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-model-cache\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.087891    4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-tmp-dir\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.087905    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26fpf\" (UniqueName: \"kubernetes.io/projected/f886013f-7c33-4731-84e8-4dcff7f18fd3-kube-api-access-26fpf\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.087922    4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-tmp-dir\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.087952    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq2pv\" (UniqueName: \"kubernetes.io/projected/29c562c6-1c99-4494-9436-137188543b90-kube-api-access-gq2pv\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.087965    4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-home\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.087978    4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-dshm\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.087991    4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-model-cache\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.088004    4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c562c6-1c99-4494-9436-137188543b90-tls-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.088016    4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-home\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.088028    4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f886013f-7c33-4731-84e8-4dcff7f18fd3-tls-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.088041    4522 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-home\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.088053    4522 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b974f0-872d-457a-b442-71e652a012cc-tls-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.088066    4522 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f886013f-7c33-4731-84e8-4dcff7f18fd3-model-cache\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.088078    4522 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b974f0-872d-457a-b442-71e652a012cc-dshm\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.088089    4522 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c562c6-1c99-4494-9436-137188543b90-tmp-dir\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.089498 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.088101    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvb89\" (UniqueName: \"kubernetes.io/projected/76b974f0-872d-457a-b442-71e652a012cc-kube-api-access-pvb89\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:22.099416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.096179    4522 generic.go:356] "Generic (PLEG): container finished" podID="76b974f0-872d-457a-b442-71e652a012cc" containerID="5f081f824c8cac562cfe1f79d493132dca3d84b12983a511986e28b363f8a861" exitCode=0
Oct 30 13:10:22.099416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.096282    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" event={"ID":"76b974f0-872d-457a-b442-71e652a012cc","Type":"ContainerDied","Data":"5f081f824c8cac562cfe1f79d493132dca3d84b12983a511986e28b363f8a861"}
Oct 30 13:10:22.099416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.096320    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n" event={"ID":"76b974f0-872d-457a-b442-71e652a012cc","Type":"ContainerDied","Data":"5f7dd620cc66812ef5d112c14fe62ffaffc62015a490bc8be51c9811642dfd1c"}
Oct 30 13:10:22.099416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.096342    4522 scope.go:122] "RemoveContainer" containerID="5f081f824c8cac562cfe1f79d493132dca3d84b12983a511986e28b363f8a861"
Oct 30 13:10:22.099416 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.096556    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"
Oct 30 13:10:22.107039 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.103859    4522 generic.go:356] "Generic (PLEG): container finished" podID="29c562c6-1c99-4494-9436-137188543b90" containerID="fd6262173ff2c2ddab9746adf517d6dbc7a7ce879d203c8e760827673abd7409" exitCode=0
Oct 30 13:10:22.107039 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.103936    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" event={"ID":"29c562c6-1c99-4494-9436-137188543b90","Type":"ContainerDied","Data":"fd6262173ff2c2ddab9746adf517d6dbc7a7ce879d203c8e760827673abd7409"}
Oct 30 13:10:22.107039 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.104050    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8" event={"ID":"29c562c6-1c99-4494-9436-137188543b90","Type":"ContainerDied","Data":"4268ef896f413d36f1efceb3c3e674e89e8c31a62f3573d77e66522ed4b41327"}
Oct 30 13:10:22.107039 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.104125    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"
Oct 30 13:10:22.108929 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.108808    4522 generic.go:356] "Generic (PLEG): container finished" podID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerID="2be2447f716cab56852b14f7e3e5e3882c73fdf42ffc53caa75b3fda08dfa752" exitCode=0
Oct 30 13:10:22.108929 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.108885    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" event={"ID":"f886013f-7c33-4731-84e8-4dcff7f18fd3","Type":"ContainerDied","Data":"2be2447f716cab56852b14f7e3e5e3882c73fdf42ffc53caa75b3fda08dfa752"}
Oct 30 13:10:22.109142 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.108937    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll" event={"ID":"f886013f-7c33-4731-84e8-4dcff7f18fd3","Type":"ContainerDied","Data":"59296e9efe7cca1289cf05d31e01d33a2337df582fa72065f5b1761db86b2c62"}
Oct 30 13:10:22.109142 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.109000    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"
Oct 30 13:10:22.110532 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.110516    4522 scope.go:122] "RemoveContainer" containerID="5f081f824c8cac562cfe1f79d493132dca3d84b12983a511986e28b363f8a861"
Oct 30 13:10:22.110866 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:10:22.110839    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f081f824c8cac562cfe1f79d493132dca3d84b12983a511986e28b363f8a861\": container with ID starting with 5f081f824c8cac562cfe1f79d493132dca3d84b12983a511986e28b363f8a861 not found: ID does not exist" containerID="5f081f824c8cac562cfe1f79d493132dca3d84b12983a511986e28b363f8a861"
Oct 30 13:10:22.110959 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.110880    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f081f824c8cac562cfe1f79d493132dca3d84b12983a511986e28b363f8a861"} err="failed to get container status \"5f081f824c8cac562cfe1f79d493132dca3d84b12983a511986e28b363f8a861\": rpc error: code = NotFound desc = could not find container \"5f081f824c8cac562cfe1f79d493132dca3d84b12983a511986e28b363f8a861\": container with ID starting with 5f081f824c8cac562cfe1f79d493132dca3d84b12983a511986e28b363f8a861 not found: ID does not exist"
Oct 30 13:10:22.110959 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.110915    4522 scope.go:122] "RemoveContainer" containerID="fd6262173ff2c2ddab9746adf517d6dbc7a7ce879d203c8e760827673abd7409"
Oct 30 13:10:22.123557 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.123532    4522 scope.go:122] "RemoveContainer" containerID="fd6262173ff2c2ddab9746adf517d6dbc7a7ce879d203c8e760827673abd7409"
Oct 30 13:10:22.123943 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:10:22.123914    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6262173ff2c2ddab9746adf517d6dbc7a7ce879d203c8e760827673abd7409\": container with ID starting with fd6262173ff2c2ddab9746adf517d6dbc7a7ce879d203c8e760827673abd7409 not found: ID does not exist" containerID="fd6262173ff2c2ddab9746adf517d6dbc7a7ce879d203c8e760827673abd7409"
Oct 30 13:10:22.124033 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.123957    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6262173ff2c2ddab9746adf517d6dbc7a7ce879d203c8e760827673abd7409"} err="failed to get container status \"fd6262173ff2c2ddab9746adf517d6dbc7a7ce879d203c8e760827673abd7409\": rpc error: code = NotFound desc = could not find container \"fd6262173ff2c2ddab9746adf517d6dbc7a7ce879d203c8e760827673abd7409\": container with ID starting with fd6262173ff2c2ddab9746adf517d6dbc7a7ce879d203c8e760827673abd7409 not found: ID does not exist"
Oct 30 13:10:22.124033 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.123986    4522 scope.go:122] "RemoveContainer" containerID="2be2447f716cab56852b14f7e3e5e3882c73fdf42ffc53caa75b3fda08dfa752"
Oct 30 13:10:22.133727 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.133701    4522 scope.go:122] "RemoveContainer" containerID="2be2447f716cab56852b14f7e3e5e3882c73fdf42ffc53caa75b3fda08dfa752"
Oct 30 13:10:22.134092 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:10:22.134032    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be2447f716cab56852b14f7e3e5e3882c73fdf42ffc53caa75b3fda08dfa752\": container with ID starting with 2be2447f716cab56852b14f7e3e5e3882c73fdf42ffc53caa75b3fda08dfa752 not found: ID does not exist" containerID="2be2447f716cab56852b14f7e3e5e3882c73fdf42ffc53caa75b3fda08dfa752"
Oct 30 13:10:22.134092 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.134060    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be2447f716cab56852b14f7e3e5e3882c73fdf42ffc53caa75b3fda08dfa752"} err="failed to get container status \"2be2447f716cab56852b14f7e3e5e3882c73fdf42ffc53caa75b3fda08dfa752\": rpc error: code = NotFound desc = could not find container \"2be2447f716cab56852b14f7e3e5e3882c73fdf42ffc53caa75b3fda08dfa752\": container with ID starting with 2be2447f716cab56852b14f7e3e5e3882c73fdf42ffc53caa75b3fda08dfa752 not found: ID does not exist"
Oct 30 13:10:22.149215 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.149165    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"]
Oct 30 13:10:22.150996 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.150971    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/simulator-w4-53dcb1-kserve-67f786dcf-fsr2n"]
Oct 30 13:10:22.216117 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.216082    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"]
Oct 30 13:10:22.225186 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.225161    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/distinct-2-w4-53dcb1-kserve-77fbf59766-8b4p8"]
Oct 30 13:10:22.261601 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.261566    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"]
Oct 30 13:10:22.267605 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.267579    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-53dcb1/premium-w4-53dcb1-kserve-654f454598-bngll"]
Oct 30 13:10:22.334458 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.334414    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c562c6-1c99-4494-9436-137188543b90" path="/var/lib/kubelet/pods/29c562c6-1c99-4494-9436-137188543b90/volumes"
Oct 30 13:10:22.334943 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.334922    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b974f0-872d-457a-b442-71e652a012cc" path="/var/lib/kubelet/pods/76b974f0-872d-457a-b442-71e652a012cc/volumes"
Oct 30 13:10:22.335392 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:22.335345    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" path="/var/lib/kubelet/pods/f886013f-7c33-4731-84e8-4dcff7f18fd3/volumes"
Oct 30 13:10:29.025449 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:29.025412    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:29.430035 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:29.429925    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"]
Oct 30 13:10:29.430232 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:29.430179    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7" podUID="051d8277-bc95-461e-92aa-ebfb4008a278" containerName="maas-api" containerID="cri-o://acbe309fd62da524c12fb5de376e6851c5c51eb0b30df4c3ed9decd89e9bbcea" gracePeriod=30
Oct 30 13:10:29.527805 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:29.527770    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:29.663065 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:29.663029    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:29.786743 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:29.786706    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:29.868475 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:29.868442    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"]
Oct 30 13:10:29.871234 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:29.871198    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53dcb1-9576n"]
Oct 30 13:10:30.168273 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.168187    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:30.272143 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.272105    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:30.333544 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.333511    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="233bf778-f71c-4b7e-b395-04b76f0a5bde" path="/var/lib/kubelet/pods/233bf778-f71c-4b7e-b395-04b76f0a5bde/volumes"
Oct 30 13:10:30.394345 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.394231    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:30.715823 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.715798    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 13:10:30.795864 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.795784    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:30.796822 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.796803    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051d8277-bc95-461e-92aa-ebfb4008a278-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051d8277-bc95-461e-92aa-ebfb4008a278-openshift-service-ca\") pod \"051d8277-bc95-461e-92aa-ebfb4008a278\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") "
Oct 30 13:10:30.796894 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.796840    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-metrics-certs\" (UniqueName: \"kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-metrics-certs\") pod \"051d8277-bc95-461e-92aa-ebfb4008a278\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") "
Oct 30 13:10:30.797036 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.797019    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051d8277-bc95-461e-92aa-ebfb4008a278-kube-api-access-4mt4x\" (UniqueName: \"kubernetes.io/projected/051d8277-bc95-461e-92aa-ebfb4008a278-kube-api-access-4mt4x\") pod \"051d8277-bc95-461e-92aa-ebfb4008a278\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") "
Oct 30 13:10:30.797110 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.797093    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-maas-api-tls\") pod \"051d8277-bc95-461e-92aa-ebfb4008a278\" (UID: \"051d8277-bc95-461e-92aa-ebfb4008a278\") "
Oct 30 13:10:30.797171 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.797149    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051d8277-bc95-461e-92aa-ebfb4008a278-openshift-service-ca" pod "051d8277-bc95-461e-92aa-ebfb4008a278" (UID: "051d8277-bc95-461e-92aa-ebfb4008a278"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:30.797452 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.797421    4522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051d8277-bc95-461e-92aa-ebfb4008a278-openshift-service-ca\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:30.799006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.798979    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-metrics-certs" pod "051d8277-bc95-461e-92aa-ebfb4008a278" (UID: "051d8277-bc95-461e-92aa-ebfb4008a278"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:30.799101 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.799046    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-maas-api-tls" pod "051d8277-bc95-461e-92aa-ebfb4008a278" (UID: "051d8277-bc95-461e-92aa-ebfb4008a278"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:30.799153 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.799118    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051d8277-bc95-461e-92aa-ebfb4008a278-kube-api-access-4mt4x" pod "051d8277-bc95-461e-92aa-ebfb4008a278" (UID: "051d8277-bc95-461e-92aa-ebfb4008a278"). InnerVolumeSpecName "kube-api-access-4mt4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:30.898312 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.898285    4522 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-metrics-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:30.898312 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.898310    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mt4x\" (UniqueName: \"kubernetes.io/projected/051d8277-bc95-461e-92aa-ebfb4008a278-kube-api-access-4mt4x\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:30.898542 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:30.898321    4522 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051d8277-bc95-461e-92aa-ebfb4008a278-maas-api-tls\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:31.178746 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:31.178659    4522 generic.go:356] "Generic (PLEG): container finished" podID="051d8277-bc95-461e-92aa-ebfb4008a278" containerID="acbe309fd62da524c12fb5de376e6851c5c51eb0b30df4c3ed9decd89e9bbcea" exitCode=0
Oct 30 13:10:31.178746 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:31.178718    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"
Oct 30 13:10:31.179240 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:31.178750    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7" event={"ID":"051d8277-bc95-461e-92aa-ebfb4008a278","Type":"ContainerDied","Data":"acbe309fd62da524c12fb5de376e6851c5c51eb0b30df4c3ed9decd89e9bbcea"}
Oct 30 13:10:31.179240 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:31.178806    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7" event={"ID":"051d8277-bc95-461e-92aa-ebfb4008a278","Type":"ContainerDied","Data":"3a7fb19e9bd53ef78fe5516becaa1fb53929b593de0fc9ceed232d2040c3a416"}
Oct 30 13:10:31.179240 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:31.178837    4522 scope.go:122] "RemoveContainer" containerID="acbe309fd62da524c12fb5de376e6851c5c51eb0b30df4c3ed9decd89e9bbcea"
Oct 30 13:10:31.188230 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:31.188213    4522 scope.go:122] "RemoveContainer" containerID="acbe309fd62da524c12fb5de376e6851c5c51eb0b30df4c3ed9decd89e9bbcea"
Oct 30 13:10:31.188480 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:10:31.188458    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acbe309fd62da524c12fb5de376e6851c5c51eb0b30df4c3ed9decd89e9bbcea\": container with ID starting with acbe309fd62da524c12fb5de376e6851c5c51eb0b30df4c3ed9decd89e9bbcea not found: ID does not exist" containerID="acbe309fd62da524c12fb5de376e6851c5c51eb0b30df4c3ed9decd89e9bbcea"
Oct 30 13:10:31.188546 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:31.188487    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acbe309fd62da524c12fb5de376e6851c5c51eb0b30df4c3ed9decd89e9bbcea"} err="failed to get container status \"acbe309fd62da524c12fb5de376e6851c5c51eb0b30df4c3ed9decd89e9bbcea\": rpc error: code = NotFound desc = could not find container \"acbe309fd62da524c12fb5de376e6851c5c51eb0b30df4c3ed9decd89e9bbcea\": container with ID starting with acbe309fd62da524c12fb5de376e6851c5c51eb0b30df4c3ed9decd89e9bbcea not found: ID does not exist"
Oct 30 13:10:31.228895 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:31.228855    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"]
Oct 30 13:10:31.240800 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:31.240775    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7a590a-86b99f7686-ptzm7"]
Oct 30 13:10:32.333081 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:32.333044    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051d8277-bc95-461e-92aa-ebfb4008a278" path="/var/lib/kubelet/pods/051d8277-bc95-461e-92aa-ebfb4008a278/volumes"
Oct 30 13:10:33.062375 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:33.062317    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-jbnbk"]
Oct 30 13:10:36.890407 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:36.890311    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:36.999246 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:36.999211    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:37.201558 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.201454    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:37.242733 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.242699    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"]
Oct 30 13:10:37.242917 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.242888    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy" containerID="cri-o://131154c300047adf1fe99173b9ace08037b0219a09f19cf581ee9b5b21b139d3" gracePeriod=30
Oct 30 13:10:37.250301 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.250274    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"]
Oct 30 13:10:37.250467 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.250447    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy" containerID="cri-o://028b8a9800f563ceeecf69d718d096218d792868f7afe945d77a03198d9fc1d9" gracePeriod=30
Oct 30 13:10:37.271928 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.271901    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"]
Oct 30 13:10:37.272100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.272082    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy" containerID="cri-o://434e2538b51212f936908b82775fb168e469cd90f5bcddbf6b46d590545d565f" gracePeriod=30
Oct 30 13:10:37.353660 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.353558    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:37.498947 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.498844    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"]
Oct 30 13:10:37.499129 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.499100    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2" podUID="c3a4b170-e0c7-4b26-8817-aa969962c106" containerName="istio-proxy" containerID="cri-o://4ad43cfedf483cf7ce66dfe8e9a830907693bc2000a3bfc501a747188a1928f2" gracePeriod=30
Oct 30 13:10:37.500810 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.500785    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"]
Oct 30 13:10:37.500948 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.500933    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" containerName="istio-proxy" containerID="cri-o://7806b84e4522665a0e3589c407fe799c9892798d05b2ff7289930a237c29a24d" gracePeriod=30
Oct 30 13:10:37.521684 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.521652    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"]
Oct 30 13:10:37.521883 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.521835    4522 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerName="istio-proxy" containerID="cri-o://7ae07040c0e0351adbd11052029b1c50dbed9f1d7a1bacacba1f4d58d1909cfd" gracePeriod=30
Oct 30 13:10:37.525779 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:37.525744    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:38.787388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:38.787330    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:10:38.787746 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:38.787417    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:10:39.709312 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:39.709279    4522 patch_prober.go:29] interesting pod/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:10:39.709492 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:39.709332    4522 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:10:40.233336 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:40.233302    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-f7d6b5c7c-jbnbk_71513e79-e472-4ad9-9c03-fd5a83fde281/limitador/0.log"
Oct 30 13:10:41.394881 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:41.394847    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-55f5889bd4-fv926_0ef93d47-69d0-4e66-9528-009fb281016d/manager/0.log"
Oct 30 13:10:42.700729 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.700702    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 13:10:42.704827 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.704804    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 13:10:42.707953 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.707934    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 13:10:42.783827 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.783759    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 13:10:42.810575 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.810549    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 13:10:42.819518 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.819498    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 13:10:42.839552 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839522    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-certs\") pod \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") "
Oct 30 13:10:42.839734 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839580    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-credential-socket\") pod \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") "
Oct 30 13:10:42.839734 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839628    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-envoy\") pod \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") "
Oct 30 13:10:42.839734 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839658    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-podinfo\") pod \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") "
Oct 30 13:10:42.839734 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839684    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-podinfo\") pod \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") "
Oct 30 13:10:42.839734 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839715    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-credential-socket\") pod \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") "
Oct 30 13:10:42.840006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839747    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istiod-ca-cert\") pod \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") "
Oct 30 13:10:42.840006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839806    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-certs\") pod \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") "
Oct 30 13:10:42.840006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839846    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istiod-ca-cert\") pod \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") "
Oct 30 13:10:42.840006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839849    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-certs" pod "88b0681f-22b7-40f0-9d42-d4efadbb44ee" (UID: "88b0681f-22b7-40f0-9d42-d4efadbb44ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.840006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839864    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-credential-socket" pod "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" (UID: "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.840006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839880    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-token\" (UniqueName: \"kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-token\") pod \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") "
Oct 30 13:10:42.840006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839918    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-kube-api-access-cbjgc\" (UniqueName: \"kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-kube-api-access-cbjgc\") pod \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") "
Oct 30 13:10:42.840006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839956    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-podinfo\") pod \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") "
Oct 30 13:10:42.840006 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.839992    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-data\") pod \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") "
Oct 30 13:10:42.840441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840004    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-credential-socket" pod "88b0681f-22b7-40f0-9d42-d4efadbb44ee" (UID: "88b0681f-22b7-40f0-9d42-d4efadbb44ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.840441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840022    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-kube-api-access-7n4fg\" (UniqueName: \"kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-kube-api-access-7n4fg\") pod \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") "
Oct 30 13:10:42.840441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840134    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-certs" pod "fe4b7a1f-8e69-4885-86ca-755ac19d97ec" (UID: "fe4b7a1f-8e69-4885-86ca-755ac19d97ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.840441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840293    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istiod-ca-cert" pod "fe4b7a1f-8e69-4885-86ca-755ac19d97ec" (UID: "fe4b7a1f-8e69-4885-86ca-755ac19d97ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:42.840441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840335    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-certs\") pod \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") "
Oct 30 13:10:42.840441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840397    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-socket\") pod \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") "
Oct 30 13:10:42.840441 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840441    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-credential-socket\") pod \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") "
Oct 30 13:10:42.840819 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840477    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-kube-api-access-jcffs\" (UniqueName: \"kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-kube-api-access-jcffs\") pod \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") "
Oct 30 13:10:42.840819 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840519    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istiod-ca-cert\") pod \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") "
Oct 30 13:10:42.840819 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840560    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-socket\") pod \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") "
Oct 30 13:10:42.840819 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840602    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-data\") pod \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") "
Oct 30 13:10:42.840819 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840655    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-envoy\") pod \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") "
Oct 30 13:10:42.840819 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840683    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-data\") pod \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") "
Oct 30 13:10:42.840819 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840722    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-socket\") pod \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\" (UID: \"88b0681f-22b7-40f0-9d42-d4efadbb44ee\") "
Oct 30 13:10:42.840819 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840753    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-token\" (UniqueName: \"kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-token\") pod \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\" (UID: \"fe4b7a1f-8e69-4885-86ca-755ac19d97ec\") "
Oct 30 13:10:42.840819 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840787    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-envoy\") pod \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") "
Oct 30 13:10:42.840819 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.840820    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-token\" (UniqueName: \"kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-token\") pod \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\" (UID: \"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc\") "
Oct 30 13:10:42.841347 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.841224    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.841347 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.841245    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.841347 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.841259    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.841347 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.841274    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.841347 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.841287    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.842195 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.841842    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-socket" pod "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" (UID: "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.842195 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.842052    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-credential-socket" pod "fe4b7a1f-8e69-4885-86ca-755ac19d97ec" (UID: "fe4b7a1f-8e69-4885-86ca-755ac19d97ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.842195 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.842189    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-socket" pod "88b0681f-22b7-40f0-9d42-d4efadbb44ee" (UID: "88b0681f-22b7-40f0-9d42-d4efadbb44ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.842480 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.842304    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-data" pod "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" (UID: "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.842480 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.842313    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-podinfo" pod "fe4b7a1f-8e69-4885-86ca-755ac19d97ec" (UID: "fe4b7a1f-8e69-4885-86ca-755ac19d97ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:42.842480 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.842319    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-certs" pod "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" (UID: "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.842480 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.842338    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-data" pod "88b0681f-22b7-40f0-9d42-d4efadbb44ee" (UID: "88b0681f-22b7-40f0-9d42-d4efadbb44ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.842701 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.842564    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-socket" pod "fe4b7a1f-8e69-4885-86ca-755ac19d97ec" (UID: "fe4b7a1f-8e69-4885-86ca-755ac19d97ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.842701 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.842607    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istiod-ca-cert" pod "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" (UID: "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:42.842701 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.842619    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-envoy" pod "88b0681f-22b7-40f0-9d42-d4efadbb44ee" (UID: "88b0681f-22b7-40f0-9d42-d4efadbb44ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.842701 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.842694    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istiod-ca-cert" pod "88b0681f-22b7-40f0-9d42-d4efadbb44ee" (UID: "88b0681f-22b7-40f0-9d42-d4efadbb44ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:42.842929 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.842826    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-data" pod "fe4b7a1f-8e69-4885-86ca-755ac19d97ec" (UID: "fe4b7a1f-8e69-4885-86ca-755ac19d97ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.843771 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.843747    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-kube-api-access-7n4fg" pod "88b0681f-22b7-40f0-9d42-d4efadbb44ee" (UID: "88b0681f-22b7-40f0-9d42-d4efadbb44ee"). InnerVolumeSpecName "kube-api-access-7n4fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.844007 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.843988    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-kube-api-access-cbjgc" pod "fe4b7a1f-8e69-4885-86ca-755ac19d97ec" (UID: "fe4b7a1f-8e69-4885-86ca-755ac19d97ec"). InnerVolumeSpecName "kube-api-access-cbjgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.844846 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.844789    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-token" pod "88b0681f-22b7-40f0-9d42-d4efadbb44ee" (UID: "88b0681f-22b7-40f0-9d42-d4efadbb44ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.845231 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.845198    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-envoy" pod "fe4b7a1f-8e69-4885-86ca-755ac19d97ec" (UID: "fe4b7a1f-8e69-4885-86ca-755ac19d97ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.845589 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.845566    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-podinfo" pod "88b0681f-22b7-40f0-9d42-d4efadbb44ee" (UID: "88b0681f-22b7-40f0-9d42-d4efadbb44ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:42.845665 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.845619    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-podinfo" pod "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" (UID: "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:42.845822 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.845796    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-envoy" pod "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" (UID: "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.845904 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.845818    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-kube-api-access-jcffs" pod "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" (UID: "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc"). InnerVolumeSpecName "kube-api-access-jcffs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.845904 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.845873    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-token" pod "fe4b7a1f-8e69-4885-86ca-755ac19d97ec" (UID: "fe4b7a1f-8e69-4885-86ca-755ac19d97ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.846291 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.846274    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-token" pod "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" (UID: "3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.941759 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.941728    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-data\") pod \"c3a4b170-e0c7-4b26-8817-aa969962c106\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") "
Oct 30 13:10:42.941759 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.941763    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-socket\") pod \"c3a4b170-e0c7-4b26-8817-aa969962c106\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") "
Oct 30 13:10:42.942100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.941785    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-podinfo\") pod \"6384f31c-d523-4d6d-ad87-e531be8bd74a\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") "
Oct 30 13:10:42.942100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.941807    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-istio-token\" (UniqueName: \"kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-istio-token\") pod \"f949644a-2fa4-48c4-a442-75846a73c62f\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") "
Oct 30 13:10:42.942100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.941837    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-certs\") pod \"f949644a-2fa4-48c4-a442-75846a73c62f\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") "
Oct 30 13:10:42.942100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.941891    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6384f31c-d523-4d6d-ad87-e531be8bd74a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6384f31c-d523-4d6d-ad87-e531be8bd74a-istiod-ca-cert\") pod \"6384f31c-d523-4d6d-ad87-e531be8bd74a\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") "
Oct 30 13:10:42.942100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.941919    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-envoy\") pod \"f949644a-2fa4-48c4-a442-75846a73c62f\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") "
Oct 30 13:10:42.942100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.941959    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-credential-socket\") pod \"c3a4b170-e0c7-4b26-8817-aa969962c106\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") "
Oct 30 13:10:42.942100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.941987    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-certs\") pod \"c3a4b170-e0c7-4b26-8817-aa969962c106\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") "
Oct 30 13:10:42.942100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.942054    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-socket" pod "c3a4b170-e0c7-4b26-8817-aa969962c106" (UID: "c3a4b170-e0c7-4b26-8817-aa969962c106"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.942100 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.942064    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-data" pod "c3a4b170-e0c7-4b26-8817-aa969962c106" (UID: "c3a4b170-e0c7-4b26-8817-aa969962c106"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.942706 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.942283    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-certs" pod "f949644a-2fa4-48c4-a442-75846a73c62f" (UID: "f949644a-2fa4-48c4-a442-75846a73c62f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.942706 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.942290    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-credential-socket" pod "c3a4b170-e0c7-4b26-8817-aa969962c106" (UID: "c3a4b170-e0c7-4b26-8817-aa969962c106"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.942706 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.942424    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-certs" pod "c3a4b170-e0c7-4b26-8817-aa969962c106" (UID: "c3a4b170-e0c7-4b26-8817-aa969962c106"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.942706 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.942515    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-credential-socket" pod "f949644a-2fa4-48c4-a442-75846a73c62f" (UID: "f949644a-2fa4-48c4-a442-75846a73c62f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.942930 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.942838    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6384f31c-d523-4d6d-ad87-e531be8bd74a-istiod-ca-cert" pod "6384f31c-d523-4d6d-ad87-e531be8bd74a" (UID: "6384f31c-d523-4d6d-ad87-e531be8bd74a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:42.942930 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.942895    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-credential-socket\") pod \"f949644a-2fa4-48c4-a442-75846a73c62f\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") "
Oct 30 13:10:42.943051 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.942958    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-data\") pod \"6384f31c-d523-4d6d-ad87-e531be8bd74a\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") "
Oct 30 13:10:42.943051 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.942997    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f949644a-2fa4-48c4-a442-75846a73c62f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f949644a-2fa4-48c4-a442-75846a73c62f-istio-podinfo\") pod \"f949644a-2fa4-48c4-a442-75846a73c62f\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") "
Oct 30 13:10:42.943051 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943040    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-data\") pod \"f949644a-2fa4-48c4-a442-75846a73c62f\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") "
Oct 30 13:10:42.943215 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943076    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-kube-api-access-ckhwx\" (UniqueName: \"kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-kube-api-access-ckhwx\") pod \"c3a4b170-e0c7-4b26-8817-aa969962c106\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") "
Oct 30 13:10:42.943215 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943114    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-envoy\") pod \"c3a4b170-e0c7-4b26-8817-aa969962c106\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") "
Oct 30 13:10:42.943323 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943272    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-credential-socket\") pod \"6384f31c-d523-4d6d-ad87-e531be8bd74a\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") "
Oct 30 13:10:42.943323 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943305    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-data" pod "f949644a-2fa4-48c4-a442-75846a73c62f" (UID: "f949644a-2fa4-48c4-a442-75846a73c62f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.943463 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943312    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-certs\") pod \"6384f31c-d523-4d6d-ad87-e531be8bd74a\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") "
Oct 30 13:10:42.943463 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943405    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-token\" (UniqueName: \"kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-token\") pod \"6384f31c-d523-4d6d-ad87-e531be8bd74a\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") "
Oct 30 13:10:42.943463 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943443    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-socket\") pod \"f949644a-2fa4-48c4-a442-75846a73c62f\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") "
Oct 30 13:10:42.943629 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943499    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-envoy\") pod \"6384f31c-d523-4d6d-ad87-e531be8bd74a\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") "
Oct 30 13:10:42.943629 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943514    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-certs" pod "6384f31c-d523-4d6d-ad87-e531be8bd74a" (UID: "6384f31c-d523-4d6d-ad87-e531be8bd74a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.943629 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943531    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-kube-api-access-vpdpf\" (UniqueName: \"kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-kube-api-access-vpdpf\") pod \"6384f31c-d523-4d6d-ad87-e531be8bd74a\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") "
Oct 30 13:10:42.943629 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943532    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-data" pod "6384f31c-d523-4d6d-ad87-e531be8bd74a" (UID: "6384f31c-d523-4d6d-ad87-e531be8bd74a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.943629 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943578    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a4b170-e0c7-4b26-8817-aa969962c106-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a4b170-e0c7-4b26-8817-aa969962c106-istiod-ca-cert\") pod \"c3a4b170-e0c7-4b26-8817-aa969962c106\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") "
Oct 30 13:10:42.943629 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943615    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-podinfo\") pod \"c3a4b170-e0c7-4b26-8817-aa969962c106\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") "
Oct 30 13:10:42.943948 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943644    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-kube-api-access-2s4bl\" (UniqueName: \"kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-kube-api-access-2s4bl\") pod \"f949644a-2fa4-48c4-a442-75846a73c62f\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") "
Oct 30 13:10:42.943948 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943674    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-token\" (UniqueName: \"kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-token\") pod \"c3a4b170-e0c7-4b26-8817-aa969962c106\" (UID: \"c3a4b170-e0c7-4b26-8817-aa969962c106\") "
Oct 30 13:10:42.943948 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943699    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-socket\") pod \"6384f31c-d523-4d6d-ad87-e531be8bd74a\" (UID: \"6384f31c-d523-4d6d-ad87-e531be8bd74a\") "
Oct 30 13:10:42.943948 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943751    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-credential-socket" pod "6384f31c-d523-4d6d-ad87-e531be8bd74a" (UID: "6384f31c-d523-4d6d-ad87-e531be8bd74a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.943948 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.943753    4522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f949644a-2fa4-48c4-a442-75846a73c62f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f949644a-2fa4-48c4-a442-75846a73c62f-istiod-ca-cert\") pod \"f949644a-2fa4-48c4-a442-75846a73c62f\" (UID: \"f949644a-2fa4-48c4-a442-75846a73c62f\") "
Oct 30 13:10:42.944246 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944000    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a4b170-e0c7-4b26-8817-aa969962c106-istiod-ca-cert" pod "c3a4b170-e0c7-4b26-8817-aa969962c106" (UID: "c3a4b170-e0c7-4b26-8817-aa969962c106"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:42.944246 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944199    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944246 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944221    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944246 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944234    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbjgc\" (UniqueName: \"kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-kube-api-access-cbjgc\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944250    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944265    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944280    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944293    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944308    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n4fg\" (UniqueName: \"kubernetes.io/projected/88b0681f-22b7-40f0-9d42-d4efadbb44ee-kube-api-access-7n4fg\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944322    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944338    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944370    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944387    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944402    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcffs\" (UniqueName: \"kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-kube-api-access-jcffs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944418    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944432    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944444    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6384f31c-d523-4d6d-ad87-e531be8bd74a-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944458    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.944470 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944471    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944484    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944499    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944512    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944527    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944540    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944553    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944566    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944579    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944593    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944607    4522 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-data\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944620    4522 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-credential-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944633    4522 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-certs\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944647    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b0681f-22b7-40f0-9d42-d4efadbb44ee-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944659    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944674    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4b7a1f-8e69-4885-86ca-755ac19d97ec-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944688    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a4b170-e0c7-4b26-8817-aa969962c106-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.945018    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-podinfo" pod "6384f31c-d523-4d6d-ad87-e531be8bd74a" (UID: "6384f31c-d523-4d6d-ad87-e531be8bd74a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:42.945256 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.945110    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-istio-token" pod "f949644a-2fa4-48c4-a442-75846a73c62f" (UID: "f949644a-2fa4-48c4-a442-75846a73c62f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.946293 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.945437    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-socket" pod "f949644a-2fa4-48c4-a442-75846a73c62f" (UID: "f949644a-2fa4-48c4-a442-75846a73c62f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.946293 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.944004    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f949644a-2fa4-48c4-a442-75846a73c62f-istiod-ca-cert" pod "f949644a-2fa4-48c4-a442-75846a73c62f" (UID: "f949644a-2fa4-48c4-a442-75846a73c62f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:42.946293 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.945801    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-envoy" pod "c3a4b170-e0c7-4b26-8817-aa969962c106" (UID: "c3a4b170-e0c7-4b26-8817-aa969962c106"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.946293 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.945969    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-envoy" pod "f949644a-2fa4-48c4-a442-75846a73c62f" (UID: "f949644a-2fa4-48c4-a442-75846a73c62f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.946293 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.946033    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-socket" pod "6384f31c-d523-4d6d-ad87-e531be8bd74a" (UID: "6384f31c-d523-4d6d-ad87-e531be8bd74a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.946583 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.946347    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-envoy" pod "6384f31c-d523-4d6d-ad87-e531be8bd74a" (UID: "6384f31c-d523-4d6d-ad87-e531be8bd74a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:42.946816 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.946789    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-kube-api-access-vpdpf" pod "6384f31c-d523-4d6d-ad87-e531be8bd74a" (UID: "6384f31c-d523-4d6d-ad87-e531be8bd74a"). InnerVolumeSpecName "kube-api-access-vpdpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.947919 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.947894    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-podinfo" pod "c3a4b170-e0c7-4b26-8817-aa969962c106" (UID: "c3a4b170-e0c7-4b26-8817-aa969962c106"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:42.947919 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.947894    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-kube-api-access-2s4bl" pod "f949644a-2fa4-48c4-a442-75846a73c62f" (UID: "f949644a-2fa4-48c4-a442-75846a73c62f"). InnerVolumeSpecName "kube-api-access-2s4bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.948088 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.947920    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-kube-api-access-ckhwx" pod "c3a4b170-e0c7-4b26-8817-aa969962c106" (UID: "c3a4b170-e0c7-4b26-8817-aa969962c106"). InnerVolumeSpecName "kube-api-access-ckhwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.948714 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.948688    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-token" pod "c3a4b170-e0c7-4b26-8817-aa969962c106" (UID: "c3a4b170-e0c7-4b26-8817-aa969962c106"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:42.948891 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.948872    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f949644a-2fa4-48c4-a442-75846a73c62f-istio-podinfo" pod "f949644a-2fa4-48c4-a442-75846a73c62f" (UID: "f949644a-2fa4-48c4-a442-75846a73c62f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:42.949201 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:42.949179    4522 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-token" pod "6384f31c-d523-4d6d-ad87-e531be8bd74a" (UID: "6384f31c-d523-4d6d-ad87-e531be8bd74a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:43.046321 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046238    4522 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f949644a-2fa4-48c4-a442-75846a73c62f-istiod-ca-cert\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046321 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046285    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046321 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046300    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046321 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046314    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046327    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f949644a-2fa4-48c4-a442-75846a73c62f-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046341    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckhwx\" (UniqueName: \"kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-kube-api-access-ckhwx\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046370    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046384    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046396    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f949644a-2fa4-48c4-a442-75846a73c62f-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046408    4522 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-istio-envoy\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046422    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpdpf\" (UniqueName: \"kubernetes.io/projected/6384f31c-d523-4d6d-ad87-e531be8bd74a-kube-api-access-vpdpf\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046435    4522 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-podinfo\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046450    4522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s4bl\" (UniqueName: \"kubernetes.io/projected/f949644a-2fa4-48c4-a442-75846a73c62f-kube-api-access-2s4bl\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046463    4522 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a4b170-e0c7-4b26-8817-aa969962c106-istio-token\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.046587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.046476    4522 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6384f31c-d523-4d6d-ad87-e531be8bd74a-workload-socket\") on node \"ip-10-0-129-133.ec2.internal\" DevicePath \"\""
Oct 30 13:10:43.237576 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.237535    4522 generic.go:356] "Generic (PLEG): container finished" podID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerID="028b8a9800f563ceeecf69d718d096218d792868f7afe945d77a03198d9fc1d9" exitCode=0
Oct 30 13:10:43.237774 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.237602    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"
Oct 30 13:10:43.237774 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.237617    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" event={"ID":"fe4b7a1f-8e69-4885-86ca-755ac19d97ec","Type":"ContainerDied","Data":"028b8a9800f563ceeecf69d718d096218d792868f7afe945d77a03198d9fc1d9"}
Oct 30 13:10:43.237774 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.237721    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm" event={"ID":"fe4b7a1f-8e69-4885-86ca-755ac19d97ec","Type":"ContainerDied","Data":"6344a427a94a5b0bccb010ec6714927d3a43f49b11d588f25e9754167b2dbb45"}
Oct 30 13:10:43.237774 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.237742    4522 scope.go:122] "RemoveContainer" containerID="028b8a9800f563ceeecf69d718d096218d792868f7afe945d77a03198d9fc1d9"
Oct 30 13:10:43.246482 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.246454    4522 generic.go:356] "Generic (PLEG): container finished" podID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerID="434e2538b51212f936908b82775fb168e469cd90f5bcddbf6b46d590545d565f" exitCode=0
Oct 30 13:10:43.246737 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.246555    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" event={"ID":"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc","Type":"ContainerDied","Data":"434e2538b51212f936908b82775fb168e469cd90f5bcddbf6b46d590545d565f"}
Oct 30 13:10:43.246737 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.246608    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5" event={"ID":"3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc","Type":"ContainerDied","Data":"763451398dcad89c4187748f47fb92a68d4ac0c199a86e93a13db3ef02e604a7"}
Oct 30 13:10:43.246737 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.246706    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"
Oct 30 13:10:43.249597 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.248940    4522 generic.go:356] "Generic (PLEG): container finished" podID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerID="7ae07040c0e0351adbd11052029b1c50dbed9f1d7a1bacacba1f4d58d1909cfd" exitCode=0
Oct 30 13:10:43.249597 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.249022    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" event={"ID":"6384f31c-d523-4d6d-ad87-e531be8bd74a","Type":"ContainerDied","Data":"7ae07040c0e0351adbd11052029b1c50dbed9f1d7a1bacacba1f4d58d1909cfd"}
Oct 30 13:10:43.249597 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.249064    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9" event={"ID":"6384f31c-d523-4d6d-ad87-e531be8bd74a","Type":"ContainerDied","Data":"bf657b6fde0eafa96b6ea7c38412da6573646c4d0c51aa38c78c8b56a9f5eebd"}
Oct 30 13:10:43.249597 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.249159    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"
Oct 30 13:10:43.251209 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.251183    4522 generic.go:356] "Generic (PLEG): container finished" podID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerID="131154c300047adf1fe99173b9ace08037b0219a09f19cf581ee9b5b21b139d3" exitCode=0
Oct 30 13:10:43.251320 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.251254    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" event={"ID":"88b0681f-22b7-40f0-9d42-d4efadbb44ee","Type":"ContainerDied","Data":"131154c300047adf1fe99173b9ace08037b0219a09f19cf581ee9b5b21b139d3"}
Oct 30 13:10:43.251320 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.251291    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv" event={"ID":"88b0681f-22b7-40f0-9d42-d4efadbb44ee","Type":"ContainerDied","Data":"52410049f90c60cf881758e476d01be99a41e18c994db21d7c6eed1719dd69dd"}
Oct 30 13:10:43.251464 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.251380    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"
Oct 30 13:10:43.254928 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.254898    4522 scope.go:122] "RemoveContainer" containerID="028b8a9800f563ceeecf69d718d096218d792868f7afe945d77a03198d9fc1d9"
Oct 30 13:10:43.256052 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:10:43.255728    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028b8a9800f563ceeecf69d718d096218d792868f7afe945d77a03198d9fc1d9\": container with ID starting with 028b8a9800f563ceeecf69d718d096218d792868f7afe945d77a03198d9fc1d9 not found: ID does not exist" containerID="028b8a9800f563ceeecf69d718d096218d792868f7afe945d77a03198d9fc1d9"
Oct 30 13:10:43.256052 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.255770    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028b8a9800f563ceeecf69d718d096218d792868f7afe945d77a03198d9fc1d9"} err="failed to get container status \"028b8a9800f563ceeecf69d718d096218d792868f7afe945d77a03198d9fc1d9\": rpc error: code = NotFound desc = could not find container \"028b8a9800f563ceeecf69d718d096218d792868f7afe945d77a03198d9fc1d9\": container with ID starting with 028b8a9800f563ceeecf69d718d096218d792868f7afe945d77a03198d9fc1d9 not found: ID does not exist"
Oct 30 13:10:43.256052 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.255793    4522 scope.go:122] "RemoveContainer" containerID="434e2538b51212f936908b82775fb168e469cd90f5bcddbf6b46d590545d565f"
Oct 30 13:10:43.258471 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.258448    4522 generic.go:356] "Generic (PLEG): container finished" podID="c3a4b170-e0c7-4b26-8817-aa969962c106" containerID="4ad43cfedf483cf7ce66dfe8e9a830907693bc2000a3bfc501a747188a1928f2" exitCode=0
Oct 30 13:10:43.258666 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.258648    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2" event={"ID":"c3a4b170-e0c7-4b26-8817-aa969962c106","Type":"ContainerDied","Data":"4ad43cfedf483cf7ce66dfe8e9a830907693bc2000a3bfc501a747188a1928f2"}
Oct 30 13:10:43.258829 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.258811    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2" event={"ID":"c3a4b170-e0c7-4b26-8817-aa969962c106","Type":"ContainerDied","Data":"f88b2138b7fce40d0550e4d45fa026a2860342e2193ccac4ac37a70ebe4de408"}
Oct 30 13:10:43.259035 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.259016    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"
Oct 30 13:10:43.261045 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.261022    4522 generic.go:356] "Generic (PLEG): container finished" podID="f949644a-2fa4-48c4-a442-75846a73c62f" containerID="7806b84e4522665a0e3589c407fe799c9892798d05b2ff7289930a237c29a24d" exitCode=0
Oct 30 13:10:43.261145 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.261128    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" event={"ID":"f949644a-2fa4-48c4-a442-75846a73c62f","Type":"ContainerDied","Data":"7806b84e4522665a0e3589c407fe799c9892798d05b2ff7289930a237c29a24d"}
Oct 30 13:10:43.261205 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.261173    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww" event={"ID":"f949644a-2fa4-48c4-a442-75846a73c62f","Type":"ContainerDied","Data":"3c3326505a137117237709f264af56504d2ef36bee67cb81889c881db4b4079d"}
Oct 30 13:10:43.261278 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.261264    4522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"
Oct 30 13:10:43.272758 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.272735    4522 scope.go:122] "RemoveContainer" containerID="3f884ee20ba3ffb2b95a8d49b5d9776fe6132a9bc34156ab2377b68194825847"
Oct 30 13:10:43.313147 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.313123    4522 scope.go:122] "RemoveContainer" containerID="434e2538b51212f936908b82775fb168e469cd90f5bcddbf6b46d590545d565f"
Oct 30 13:10:43.313480 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:10:43.313459    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434e2538b51212f936908b82775fb168e469cd90f5bcddbf6b46d590545d565f\": container with ID starting with 434e2538b51212f936908b82775fb168e469cd90f5bcddbf6b46d590545d565f not found: ID does not exist" containerID="434e2538b51212f936908b82775fb168e469cd90f5bcddbf6b46d590545d565f"
Oct 30 13:10:43.313564 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.313498    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434e2538b51212f936908b82775fb168e469cd90f5bcddbf6b46d590545d565f"} err="failed to get container status \"434e2538b51212f936908b82775fb168e469cd90f5bcddbf6b46d590545d565f\": rpc error: code = NotFound desc = could not find container \"434e2538b51212f936908b82775fb168e469cd90f5bcddbf6b46d590545d565f\": container with ID starting with 434e2538b51212f936908b82775fb168e469cd90f5bcddbf6b46d590545d565f not found: ID does not exist"
Oct 30 13:10:43.313564 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.313515    4522 scope.go:122] "RemoveContainer" containerID="3f884ee20ba3ffb2b95a8d49b5d9776fe6132a9bc34156ab2377b68194825847"
Oct 30 13:10:43.313783 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:10:43.313767    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f884ee20ba3ffb2b95a8d49b5d9776fe6132a9bc34156ab2377b68194825847\": container with ID starting with 3f884ee20ba3ffb2b95a8d49b5d9776fe6132a9bc34156ab2377b68194825847 not found: ID does not exist" containerID="3f884ee20ba3ffb2b95a8d49b5d9776fe6132a9bc34156ab2377b68194825847"
Oct 30 13:10:43.313831 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.313787    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f884ee20ba3ffb2b95a8d49b5d9776fe6132a9bc34156ab2377b68194825847"} err="failed to get container status \"3f884ee20ba3ffb2b95a8d49b5d9776fe6132a9bc34156ab2377b68194825847\": rpc error: code = NotFound desc = could not find container \"3f884ee20ba3ffb2b95a8d49b5d9776fe6132a9bc34156ab2377b68194825847\": container with ID starting with 3f884ee20ba3ffb2b95a8d49b5d9776fe6132a9bc34156ab2377b68194825847 not found: ID does not exist"
Oct 30 13:10:43.313831 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.313800    4522 scope.go:122] "RemoveContainer" containerID="7ae07040c0e0351adbd11052029b1c50dbed9f1d7a1bacacba1f4d58d1909cfd"
Oct 30 13:10:43.322287 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.322270    4522 scope.go:122] "RemoveContainer" containerID="7ae07040c0e0351adbd11052029b1c50dbed9f1d7a1bacacba1f4d58d1909cfd"
Oct 30 13:10:43.322546 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:10:43.322527    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae07040c0e0351adbd11052029b1c50dbed9f1d7a1bacacba1f4d58d1909cfd\": container with ID starting with 7ae07040c0e0351adbd11052029b1c50dbed9f1d7a1bacacba1f4d58d1909cfd not found: ID does not exist" containerID="7ae07040c0e0351adbd11052029b1c50dbed9f1d7a1bacacba1f4d58d1909cfd"
Oct 30 13:10:43.322610 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.322555    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae07040c0e0351adbd11052029b1c50dbed9f1d7a1bacacba1f4d58d1909cfd"} err="failed to get container status \"7ae07040c0e0351adbd11052029b1c50dbed9f1d7a1bacacba1f4d58d1909cfd\": rpc error: code = NotFound desc = could not find container \"7ae07040c0e0351adbd11052029b1c50dbed9f1d7a1bacacba1f4d58d1909cfd\": container with ID starting with 7ae07040c0e0351adbd11052029b1c50dbed9f1d7a1bacacba1f4d58d1909cfd not found: ID does not exist"
Oct 30 13:10:43.322610 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.322570    4522 scope.go:122] "RemoveContainer" containerID="131154c300047adf1fe99173b9ace08037b0219a09f19cf581ee9b5b21b139d3"
Oct 30 13:10:43.330394 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.330376    4522 scope.go:122] "RemoveContainer" containerID="131154c300047adf1fe99173b9ace08037b0219a09f19cf581ee9b5b21b139d3"
Oct 30 13:10:43.330654 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:10:43.330635    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131154c300047adf1fe99173b9ace08037b0219a09f19cf581ee9b5b21b139d3\": container with ID starting with 131154c300047adf1fe99173b9ace08037b0219a09f19cf581ee9b5b21b139d3 not found: ID does not exist" containerID="131154c300047adf1fe99173b9ace08037b0219a09f19cf581ee9b5b21b139d3"
Oct 30 13:10:43.330722 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.330660    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131154c300047adf1fe99173b9ace08037b0219a09f19cf581ee9b5b21b139d3"} err="failed to get container status \"131154c300047adf1fe99173b9ace08037b0219a09f19cf581ee9b5b21b139d3\": rpc error: code = NotFound desc = could not find container \"131154c300047adf1fe99173b9ace08037b0219a09f19cf581ee9b5b21b139d3\": container with ID starting with 131154c300047adf1fe99173b9ace08037b0219a09f19cf581ee9b5b21b139d3 not found: ID does not exist"
Oct 30 13:10:43.330722 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.330675    4522 scope.go:122] "RemoveContainer" containerID="4ad43cfedf483cf7ce66dfe8e9a830907693bc2000a3bfc501a747188a1928f2"
Oct 30 13:10:43.331686 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.331669    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:43.339148 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.339104    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"]
Oct 30 13:10:43.339204 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.339185    4522 scope.go:122] "RemoveContainer" containerID="4ad43cfedf483cf7ce66dfe8e9a830907693bc2000a3bfc501a747188a1928f2"
Oct 30 13:10:43.339472 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:10:43.339447    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad43cfedf483cf7ce66dfe8e9a830907693bc2000a3bfc501a747188a1928f2\": container with ID starting with 4ad43cfedf483cf7ce66dfe8e9a830907693bc2000a3bfc501a747188a1928f2 not found: ID does not exist" containerID="4ad43cfedf483cf7ce66dfe8e9a830907693bc2000a3bfc501a747188a1928f2"
Oct 30 13:10:43.339562 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.339480    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad43cfedf483cf7ce66dfe8e9a830907693bc2000a3bfc501a747188a1928f2"} err="failed to get container status \"4ad43cfedf483cf7ce66dfe8e9a830907693bc2000a3bfc501a747188a1928f2\": rpc error: code = NotFound desc = could not find container \"4ad43cfedf483cf7ce66dfe8e9a830907693bc2000a3bfc501a747188a1928f2\": container with ID starting with 4ad43cfedf483cf7ce66dfe8e9a830907693bc2000a3bfc501a747188a1928f2 not found: ID does not exist"
Oct 30 13:10:43.339562 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.339496    4522 scope.go:122] "RemoveContainer" containerID="7806b84e4522665a0e3589c407fe799c9892798d05b2ff7289930a237c29a24d"
Oct 30 13:10:43.343117 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.343083    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-dvzsm"]
Oct 30 13:10:43.348147 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.348128    4522 scope.go:122] "RemoveContainer" containerID="7806b84e4522665a0e3589c407fe799c9892798d05b2ff7289930a237c29a24d"
Oct 30 13:10:43.348423 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:10:43.348402    4522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7806b84e4522665a0e3589c407fe799c9892798d05b2ff7289930a237c29a24d\": container with ID starting with 7806b84e4522665a0e3589c407fe799c9892798d05b2ff7289930a237c29a24d not found: ID does not exist" containerID="7806b84e4522665a0e3589c407fe799c9892798d05b2ff7289930a237c29a24d"
Oct 30 13:10:43.348490 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.348432    4522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7806b84e4522665a0e3589c407fe799c9892798d05b2ff7289930a237c29a24d"} err="failed to get container status \"7806b84e4522665a0e3589c407fe799c9892798d05b2ff7289930a237c29a24d\": rpc error: code = NotFound desc = could not find container \"7806b84e4522665a0e3589c407fe799c9892798d05b2ff7289930a237c29a24d\": container with ID starting with 7806b84e4522665a0e3589c407fe799c9892798d05b2ff7289930a237c29a24d not found: ID does not exist"
Oct 30 13:10:43.393818 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.393751    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"]
Oct 30 13:10:43.396616 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.396581    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-gjqww"]
Oct 30 13:10:43.430338 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.430305    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"]
Oct 30 13:10:43.436476 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.436448    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-xdqm9"]
Oct 30 13:10:43.439413 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.439389    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:43.471545 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.471512    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"]
Oct 30 13:10:43.476390 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.476336    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a590a-openshift-default-7d55f4c4df-6j7g2"]
Oct 30 13:10:43.512773 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.512737    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"]
Oct 30 13:10:43.523042 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.523006    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-g8qd5"]
Oct 30 13:10:43.562165 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.562082    4522 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"]
Oct 30 13:10:43.566824 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.566798    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:43.567224 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.567205    4522 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-53dcb1-openshift-default-5d7f9b4f49-5nbfv"]
Oct 30 13:10:43.664440 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.664402    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:43.795974 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:43.795944    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:44.061077 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:44.061047    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:44.333343 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:44.333265    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" path="/var/lib/kubelet/pods/3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc/volumes"
Oct 30 13:10:44.333740 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:44.333725    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" path="/var/lib/kubelet/pods/6384f31c-d523-4d6d-ad87-e531be8bd74a/volumes"
Oct 30 13:10:44.334101 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:44.334089    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" path="/var/lib/kubelet/pods/88b0681f-22b7-40f0-9d42-d4efadbb44ee/volumes"
Oct 30 13:10:44.334782 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:44.334767    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a4b170-e0c7-4b26-8817-aa969962c106" path="/var/lib/kubelet/pods/c3a4b170-e0c7-4b26-8817-aa969962c106/volumes"
Oct 30 13:10:44.335147 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:44.335135    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" path="/var/lib/kubelet/pods/f949644a-2fa4-48c4-a442-75846a73c62f/volumes"
Oct 30 13:10:44.335510 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:44.335499    4522 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" path="/var/lib/kubelet/pods/fe4b7a1f-8e69-4885-86ca-755ac19d97ec/volumes"
Oct 30 13:10:45.519636 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:45.519600    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:45.731921 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:45.731892    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:46.040869 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:46.040837    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:46.142622 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:46.142583    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:50.587657 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:50.587622    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:51.134316 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:10:51.134282    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:10:51.346221 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:51.346186    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:51.666157 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:51.666124    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:54.434602 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:54.434566    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-55f5889bd4-fv926_0ef93d47-69d0-4e66-9528-009fb281016d/manager/0.log"
Oct 30 13:10:57.227736 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:57.227704    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0315ecbb-n48gd_165a54eb-a4c8-4193-9d01-952415abb48f/revoke-keys/0.log"
Oct 30 13:10:57.414803 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:57.414770    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0f9d9a42-dcvjg_db0791c0-56d1-427d-a1a5-2ec1f1af434b/revoke-keys/0.log"
Oct 30 13:10:57.761457 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:57.761429    4522 kubelet.go:1689] "Image garbage collection succeeded"
Oct 30 13:10:57.947888 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:10:57.947849    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3375bbdf-rn848_a8b12332-aec4-4639-8acb-c7534329da3b/revoke-keys/0.log"
Oct 30 13:11:02.131273 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:02.131242    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-f7d6b5c7c-jbnbk_71513e79-e472-4ad9-9c03-fd5a83fde281/limitador/0.log"
Oct 30 13:11:03.611800 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:03.611765    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kfdb7_c2c7a319-32ec-4793-8d0a-2a0b60fb8d53/discovery/0.log"
Oct 30 13:11:04.005973 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:04.005889    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-dc45784b6-l4s7t_01215889-00f7-4cfd-97cd-dee00355f9b9/kube-auth-proxy/0.log"
Oct 30 13:11:04.564399 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:04.564369    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-9twbb_f352699b-bc5c-4be3-9f32-ebdec5169271/istio-proxy/0.log"
Oct 30 13:11:05.143238 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:05.143211    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-pdt6z_f7be2765-5a1f-42db-94fd-91c0349e403d/payload-pre-processing/0.log"
Oct 30 13:11:05.345549 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:05.345517    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hv6zx_4098fe5a-e7d3-488c-a1ed-72d447999e15/payload-processing/0.log"
Oct 30 13:11:06.266918 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:06.266877    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-gwwxj_9f133540-b63f-449a-8377-36912a0f1b35/main/0.log"
Oct 30 13:11:06.636070 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:06.636041    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-z7swk_29d90fc8-e584-45cb-9e65-dfaa8c7bc226/main/0.log"
Oct 30 13:11:06.983543 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:06.983460    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88rhgz_f2e6d063-6dbf-4f1b-9f07-01c668500c25/main/0.log"
Oct 30 13:11:07.517787 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:07.517761    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-hxswq_ce95115a-fa86-4eab-9f15-1f455288c014/main/0.log"
Oct 30 13:11:17.803328 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803108    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2zns2/must-gather-pg9qt"]
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803585    4522 cpu_manager.go:414] "containerMap: removing container" podUID="db0791c0-56d1-427d-a1a5-2ec1f1af434b" containerName="revoke-keys"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803602    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0791c0-56d1-427d-a1a5-2ec1f1af434b" containerName="revoke-keys"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803618    4522 cpu_manager.go:414] "containerMap: removing container" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803623    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803633    4522 cpu_manager.go:414] "containerMap: removing container" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803638    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803649    4522 cpu_manager.go:414] "containerMap: removing container" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" containerName="istio-proxy"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803654    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" containerName="istio-proxy"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803661    4522 cpu_manager.go:414] "containerMap: removing container" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803665    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803673    4522 cpu_manager.go:414] "containerMap: removing container" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803677    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803690    4522 cpu_manager.go:414] "containerMap: removing container" podUID="c3a4b170-e0c7-4b26-8817-aa969962c106" containerName="istio-proxy"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803694    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a4b170-e0c7-4b26-8817-aa969962c106" containerName="istio-proxy"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803703    4522 cpu_manager.go:414] "containerMap: removing container" podUID="051d8277-bc95-461e-92aa-ebfb4008a278" containerName="maas-api"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803708    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051d8277-bc95-461e-92aa-ebfb4008a278" containerName="maas-api"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803714    4522 cpu_manager.go:414] "containerMap: removing container" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerName="istio-proxy"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803719    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerName="istio-proxy"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803727    4522 cpu_manager.go:414] "containerMap: removing container" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803731    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803738    4522 cpu_manager.go:414] "containerMap: removing container" podUID="a8b12332-aec4-4639-8acb-c7534329da3b" containerName="revoke-keys"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803742    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b12332-aec4-4639-8acb-c7534329da3b" containerName="revoke-keys"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803749    4522 cpu_manager.go:414] "containerMap: removing container" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy"
Oct 30 13:11:17.803766 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803754    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803804    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a4b170-e0c7-4b26-8817-aa969962c106" containerName="istio-proxy"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803815    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="051d8277-bc95-461e-92aa-ebfb4008a278" containerName="maas-api"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803821    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="6384f31c-d523-4d6d-ad87-e531be8bd74a" containerName="istio-proxy"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803827    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b0681f-22b7-40f0-9d42-d4efadbb44ee" containerName="istio-proxy"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803833    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b12332-aec4-4639-8acb-c7534329da3b" containerName="revoke-keys"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803839    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe4b7a1f-8e69-4885-86ca-755ac19d97ec" containerName="istio-proxy"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803845    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803851    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0791c0-56d1-427d-a1a5-2ec1f1af434b" containerName="revoke-keys"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803857    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f886013f-7c33-4731-84e8-4dcff7f18fd3" containerName="main"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803863    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c562c6-1c99-4494-9436-137188543b90" containerName="main"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803867    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="f949644a-2fa4-48c4-a442-75846a73c62f" containerName="istio-proxy"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803872    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b974f0-872d-457a-b442-71e652a012cc" containerName="main"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803931    4522 cpu_manager.go:414] "containerMap: removing container" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.803937    4522 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy"
Oct 30 13:11:17.804631 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.804003    4522 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e41433e-dd8f-4a3b-aaa2-9f3c8aac3bcc" containerName="istio-proxy"
Oct 30 13:11:17.807025 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.807007    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2zns2/must-gather-pg9qt"
Oct 30 13:11:17.822162 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.822142    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2zns2\"/\"openshift-service-ca.crt\""
Oct 30 13:11:17.822258 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.822184    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2zns2\"/\"kube-root-ca.crt\""
Oct 30 13:11:17.834257 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.834218    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2zns2\"/\"default-dockercfg-t68tc\""
Oct 30 13:11:17.836876 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.836851    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2zns2/must-gather-pg9qt"]
Oct 30 13:11:17.848704 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.848679    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c9c8427b-aa3d-459b-b6fd-c3c950fabf54-must-gather-output\") pod \"must-gather-pg9qt\" (UID: \"c9c8427b-aa3d-459b-b6fd-c3c950fabf54\") " pod="openshift-must-gather-2zns2/must-gather-pg9qt"
Oct 30 13:11:17.848806 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.848742    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgs2p\" (UniqueName: \"kubernetes.io/projected/c9c8427b-aa3d-459b-b6fd-c3c950fabf54-kube-api-access-pgs2p\") pod \"must-gather-pg9qt\" (UID: \"c9c8427b-aa3d-459b-b6fd-c3c950fabf54\") " pod="openshift-must-gather-2zns2/must-gather-pg9qt"
Oct 30 13:11:17.871094 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.871068    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:17.896085 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.896064    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-29dh2/must-gather-jqkbg"]
Oct 30 13:11:17.899400 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.899385    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-29dh2/must-gather-jqkbg"
Oct 30 13:11:17.917640 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.917620    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-29dh2/must-gather-jqkbg"]
Oct 30 13:11:17.922595 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.922570    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-29dh2\"/\"openshift-service-ca.crt\""
Oct 30 13:11:17.922711 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.922599    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-29dh2\"/\"default-dockercfg-gnrhj\""
Oct 30 13:11:17.922711 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.922612    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-29dh2\"/\"kube-root-ca.crt\""
Oct 30 13:11:17.949467 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.949445    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c4f3ea22-b051-4913-b54f-057f434ca323-must-gather-output\") pod \"must-gather-jqkbg\" (UID: \"c4f3ea22-b051-4913-b54f-057f434ca323\") " pod="openshift-must-gather-29dh2/must-gather-jqkbg"
Oct 30 13:11:17.949557 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.949492    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhsvk\" (UniqueName: \"kubernetes.io/projected/c4f3ea22-b051-4913-b54f-057f434ca323-kube-api-access-hhsvk\") pod \"must-gather-jqkbg\" (UID: \"c4f3ea22-b051-4913-b54f-057f434ca323\") " pod="openshift-must-gather-29dh2/must-gather-jqkbg"
Oct 30 13:11:17.949557 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.949516    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c9c8427b-aa3d-459b-b6fd-c3c950fabf54-must-gather-output\") pod \"must-gather-pg9qt\" (UID: \"c9c8427b-aa3d-459b-b6fd-c3c950fabf54\") " pod="openshift-must-gather-2zns2/must-gather-pg9qt"
Oct 30 13:11:17.949635 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.949621    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgs2p\" (UniqueName: \"kubernetes.io/projected/c9c8427b-aa3d-459b-b6fd-c3c950fabf54-kube-api-access-pgs2p\") pod \"must-gather-pg9qt\" (UID: \"c9c8427b-aa3d-459b-b6fd-c3c950fabf54\") " pod="openshift-must-gather-2zns2/must-gather-pg9qt"
Oct 30 13:11:17.949764 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.949751    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c9c8427b-aa3d-459b-b6fd-c3c950fabf54-must-gather-output\") pod \"must-gather-pg9qt\" (UID: \"c9c8427b-aa3d-459b-b6fd-c3c950fabf54\") " pod="openshift-must-gather-2zns2/must-gather-pg9qt"
Oct 30 13:11:17.970203 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.970180    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgs2p\" (UniqueName: \"kubernetes.io/projected/c9c8427b-aa3d-459b-b6fd-c3c950fabf54-kube-api-access-pgs2p\") pod \"must-gather-pg9qt\" (UID: \"c9c8427b-aa3d-459b-b6fd-c3c950fabf54\") " pod="openshift-must-gather-2zns2/must-gather-pg9qt"
Oct 30 13:11:17.991652 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.991624    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:17.993041 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.993022    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5tjtq/must-gather-8jf6k"]
Oct 30 13:11:17.996388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:17.996373    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tjtq/must-gather-8jf6k"
Oct 30 13:11:18.018474 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.018452    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5tjtq\"/\"openshift-service-ca.crt\""
Oct 30 13:11:18.018586 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.018485    4522 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5tjtq\"/\"kube-root-ca.crt\""
Oct 30 13:11:18.020733 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.020718    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tjtq/must-gather-8jf6k"]
Oct 30 13:11:18.026287 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.026272    4522 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5tjtq\"/\"default-dockercfg-rwqwj\""
Oct 30 13:11:18.050874 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.050853    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlbzl\" (UniqueName: \"kubernetes.io/projected/965dd41e-d6f1-4c1b-87c5-cfd5091c9702-kube-api-access-nlbzl\") pod \"must-gather-8jf6k\" (UID: \"965dd41e-d6f1-4c1b-87c5-cfd5091c9702\") " pod="openshift-must-gather-5tjtq/must-gather-8jf6k"
Oct 30 13:11:18.050982 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.050884    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c4f3ea22-b051-4913-b54f-057f434ca323-must-gather-output\") pod \"must-gather-jqkbg\" (UID: \"c4f3ea22-b051-4913-b54f-057f434ca323\") " pod="openshift-must-gather-29dh2/must-gather-jqkbg"
Oct 30 13:11:18.050982 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.050918    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/965dd41e-d6f1-4c1b-87c5-cfd5091c9702-must-gather-output\") pod \"must-gather-8jf6k\" (UID: \"965dd41e-d6f1-4c1b-87c5-cfd5091c9702\") " pod="openshift-must-gather-5tjtq/must-gather-8jf6k"
Oct 30 13:11:18.051154 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.051047    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhsvk\" (UniqueName: \"kubernetes.io/projected/c4f3ea22-b051-4913-b54f-057f434ca323-kube-api-access-hhsvk\") pod \"must-gather-jqkbg\" (UID: \"c4f3ea22-b051-4913-b54f-057f434ca323\") " pod="openshift-must-gather-29dh2/must-gather-jqkbg"
Oct 30 13:11:18.051196 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.051166    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c4f3ea22-b051-4913-b54f-057f434ca323-must-gather-output\") pod \"must-gather-jqkbg\" (UID: \"c4f3ea22-b051-4913-b54f-057f434ca323\") " pod="openshift-must-gather-29dh2/must-gather-jqkbg"
Oct 30 13:11:18.074205 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.074153    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhsvk\" (UniqueName: \"kubernetes.io/projected/c4f3ea22-b051-4913-b54f-057f434ca323-kube-api-access-hhsvk\") pod \"must-gather-jqkbg\" (UID: \"c4f3ea22-b051-4913-b54f-057f434ca323\") " pod="openshift-must-gather-29dh2/must-gather-jqkbg"
Oct 30 13:11:18.120254 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.120234    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2zns2/must-gather-pg9qt"
Oct 30 13:11:18.151919 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.151898    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbzl\" (UniqueName: \"kubernetes.io/projected/965dd41e-d6f1-4c1b-87c5-cfd5091c9702-kube-api-access-nlbzl\") pod \"must-gather-8jf6k\" (UID: \"965dd41e-d6f1-4c1b-87c5-cfd5091c9702\") " pod="openshift-must-gather-5tjtq/must-gather-8jf6k"
Oct 30 13:11:18.152027 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.151938    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/965dd41e-d6f1-4c1b-87c5-cfd5091c9702-must-gather-output\") pod \"must-gather-8jf6k\" (UID: \"965dd41e-d6f1-4c1b-87c5-cfd5091c9702\") " pod="openshift-must-gather-5tjtq/must-gather-8jf6k"
Oct 30 13:11:18.152243 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.152228    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/965dd41e-d6f1-4c1b-87c5-cfd5091c9702-must-gather-output\") pod \"must-gather-8jf6k\" (UID: \"965dd41e-d6f1-4c1b-87c5-cfd5091c9702\") " pod="openshift-must-gather-5tjtq/must-gather-8jf6k"
Oct 30 13:11:18.171562 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.171539    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbzl\" (UniqueName: \"kubernetes.io/projected/965dd41e-d6f1-4c1b-87c5-cfd5091c9702-kube-api-access-nlbzl\") pod \"must-gather-8jf6k\" (UID: \"965dd41e-d6f1-4c1b-87c5-cfd5091c9702\") " pod="openshift-must-gather-5tjtq/must-gather-8jf6k"
Oct 30 13:11:18.211991 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.211968    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-29dh2/must-gather-jqkbg"
Oct 30 13:11:18.308994 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.308966    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tjtq/must-gather-8jf6k"
Oct 30 13:11:18.414361 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.414322    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2zns2/must-gather-pg9qt"]
Oct 30 13:11:18.414794 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:11:18.414773    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9c8427b_aa3d_459b_b6fd_c3c950fabf54.slice/crio-d54ec789a6812528fd955f6f111d51d72c4f16c7dd1f673d290c6448bcf9d243 WatchSource:0}: Error finding container d54ec789a6812528fd955f6f111d51d72c4f16c7dd1f673d290c6448bcf9d243: Status 404 returned error can't find the container with id d54ec789a6812528fd955f6f111d51d72c4f16c7dd1f673d290c6448bcf9d243
Oct 30 13:11:18.416683 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.416657    4522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:11:18.510063 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.510039    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-29dh2/must-gather-jqkbg"]
Oct 30 13:11:18.510543 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:11:18.510520    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4f3ea22_b051_4913_b54f_057f434ca323.slice/crio-ff4f2c20eb449780d7e25ea732bccfa136f52e2412ca5336b9b3f4b7e1631b81 WatchSource:0}: Error finding container ff4f2c20eb449780d7e25ea732bccfa136f52e2412ca5336b9b3f4b7e1631b81: Status 404 returned error can't find the container with id ff4f2c20eb449780d7e25ea732bccfa136f52e2412ca5336b9b3f4b7e1631b81
Oct 30 13:11:18.609847 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:18.609825    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tjtq/must-gather-8jf6k"]
Oct 30 13:11:18.610155 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:11:18.610134    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod965dd41e_d6f1_4c1b_87c5_cfd5091c9702.slice/crio-4c431db40f111be1ad87136951a0cdbd4679c39e08192140cdbc9565687400d1 WatchSource:0}: Error finding container 4c431db40f111be1ad87136951a0cdbd4679c39e08192140cdbc9565687400d1: Status 404 returned error can't find the container with id 4c431db40f111be1ad87136951a0cdbd4679c39e08192140cdbc9565687400d1
Oct 30 13:11:19.453587 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:19.453546    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-29dh2/must-gather-jqkbg" event={"ID":"c4f3ea22-b051-4913-b54f-057f434ca323","Type":"ContainerStarted","Data":"ff4f2c20eb449780d7e25ea732bccfa136f52e2412ca5336b9b3f4b7e1631b81"}
Oct 30 13:11:19.469684 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:19.469648    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2zns2/must-gather-pg9qt" event={"ID":"c9c8427b-aa3d-459b-b6fd-c3c950fabf54","Type":"ContainerStarted","Data":"d54ec789a6812528fd955f6f111d51d72c4f16c7dd1f673d290c6448bcf9d243"}
Oct 30 13:11:19.477205 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:19.477172    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tjtq/must-gather-8jf6k" event={"ID":"965dd41e-d6f1-4c1b-87c5-cfd5091c9702","Type":"ContainerStarted","Data":"4c431db40f111be1ad87136951a0cdbd4679c39e08192140cdbc9565687400d1"}
Oct 30 13:11:20.497388 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:20.497335    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tjtq/must-gather-8jf6k" event={"ID":"965dd41e-d6f1-4c1b-87c5-cfd5091c9702","Type":"ContainerStarted","Data":"89e030d5b27cee04a9b70a07bce363ad9c8d9ec0739a9c47e89b45f0656ec045"}
Oct 30 13:11:21.174220 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:11:21.174173    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:11:21.511279 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:21.511194    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tjtq/must-gather-8jf6k" event={"ID":"965dd41e-d6f1-4c1b-87c5-cfd5091c9702","Type":"ContainerStarted","Data":"1f8d554b6757f1454a689cf7558dec3383ba100630a98d168035077666555c16"}
Oct 30 13:11:21.550877 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:21.550830    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5tjtq/must-gather-8jf6k" podStartSLOduration=3.075975179 podStartE2EDuration="4.5508145s" totalImagesPullingTime="1.474839321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:21.549670006 +0000 UTC m=+2125.364996495" watchObservedRunningTime="2026-10-30 13:11:21.5508145 +0000 UTC m=+2125.366140989"
Oct 30 13:11:23.973824 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:23.973791    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9lfhr_8c94f5d7-3c75-41bd-963f-a6fffa86747a/global-pull-secret-syncer/0.log"
Oct 30 13:11:24.823081 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:24.823049    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log"
Oct 30 13:11:25.162701 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:25.162619    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-133.ec2.internal_3c78310b24bd820f4d32df7a2e462ac4/haproxy/0.log"
Oct 30 13:11:33.638105 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:33.637924    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-f7d6b5c7c-jbnbk_71513e79-e472-4ad9-9c03-fd5a83fde281/limitador/0.log"
Oct 30 13:11:33.659480 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:33.659392    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2zns2/must-gather-pg9qt" event={"ID":"c9c8427b-aa3d-459b-b6fd-c3c950fabf54","Type":"ContainerStarted","Data":"c43ec514c6ca0f3da49629520a004badf6fa597ce83077697e4e44d3be8b5528"}
Oct 30 13:11:33.659480 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:33.659447    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2zns2/must-gather-pg9qt" event={"ID":"c9c8427b-aa3d-459b-b6fd-c3c950fabf54","Type":"ContainerStarted","Data":"d24bff2d8a9a22e1302571f54c1f87b8c60b9a8796c46a035e85f4f94b08c084"}
Oct 30 13:11:33.661483 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:33.661453    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-29dh2/must-gather-jqkbg" event={"ID":"c4f3ea22-b051-4913-b54f-057f434ca323","Type":"ContainerStarted","Data":"c4d189f6679ad3495c33abc864cdac0f72e479b48ccb7e7bc7d17030112a8c54"}
Oct 30 13:11:33.661620 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:33.661490    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-29dh2/must-gather-jqkbg" event={"ID":"c4f3ea22-b051-4913-b54f-057f434ca323","Type":"ContainerStarted","Data":"88bdc52bb48b456550e12a99637d0e80f112f939f57ac9485a8c2eabda0fb8a3"}
Oct 30 13:11:33.706379 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:33.700840    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-2zns2/must-gather-pg9qt" podStartSLOduration=2.603836266 podStartE2EDuration="16.700826595s" totalImagesPullingTime="14.096990329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:33.698702684 +0000 UTC m=+2137.514029200" watchObservedRunningTime="2026-10-30 13:11:33.700826595 +0000 UTC m=+2137.516153083"
Oct 30 13:11:35.296094 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.296040    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-29dh2/must-gather-jqkbg" podStartSLOduration=4.270812927 podStartE2EDuration="18.296024411s" totalImagesPullingTime="14.025211484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:33.741575317 +0000 UTC m=+2137.556901807" watchObservedRunningTime="2026-10-30 13:11:35.296024411 +0000 UTC m=+2139.111350942"
Oct 30 13:11:35.297227 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.297203    4522 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"]
Oct 30 13:11:35.302517 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.302498    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.322762 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.322507    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"]
Oct 30 13:11:35.377786 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.377747    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vldjd\" (UniqueName: \"kubernetes.io/projected/e35f3021-173f-475c-8285-deb7716c1322-kube-api-access-vldjd\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.377952 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.377808    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e35f3021-173f-475c-8285-deb7716c1322-lib-modules\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.377952 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.377876    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e35f3021-173f-475c-8285-deb7716c1322-podres\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.377952 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.377928    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e35f3021-173f-475c-8285-deb7716c1322-proc\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.378101 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.377955    4522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e35f3021-173f-475c-8285-deb7716c1322-sys\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.479506 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.479475    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e35f3021-173f-475c-8285-deb7716c1322-proc\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.479672 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.479514    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e35f3021-173f-475c-8285-deb7716c1322-sys\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.479672 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.479595    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e35f3021-173f-475c-8285-deb7716c1322-proc\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.479672 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.479597    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vldjd\" (UniqueName: \"kubernetes.io/projected/e35f3021-173f-475c-8285-deb7716c1322-kube-api-access-vldjd\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.479672 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.479651    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e35f3021-173f-475c-8285-deb7716c1322-lib-modules\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.479672 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.479664    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e35f3021-173f-475c-8285-deb7716c1322-sys\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.479908 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.479683    4522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e35f3021-173f-475c-8285-deb7716c1322-podres\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.479908 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.479735    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e35f3021-173f-475c-8285-deb7716c1322-podres\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.479908 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.479779    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e35f3021-173f-475c-8285-deb7716c1322-lib-modules\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.496834 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.496805    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-55f5889bd4-fv926_0ef93d47-69d0-4e66-9528-009fb281016d/manager/0.log"
Oct 30 13:11:35.505641 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.505614    4522 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vldjd\" (UniqueName: \"kubernetes.io/projected/e35f3021-173f-475c-8285-deb7716c1322-kube-api-access-vldjd\") pod \"perf-node-gather-daemonset-cxppq\" (UID: \"e35f3021-173f-475c-8285-deb7716c1322\") " pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.596212 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.596144    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-55f5889bd4-fv926_0ef93d47-69d0-4e66-9528-009fb281016d/manager/0.log"
Oct 30 13:11:35.627184 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.627159    4522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:35.941320 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:35.941290    4522 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"]
Oct 30 13:11:35.949787 ip-10-0-129-133 kubenswrapper[4522]: W1030 13:11:35.949752    4522 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode35f3021_173f_475c_8285_deb7716c1322.slice/crio-f5a558d2fc8fab6830e9976cab08ada076232bfead1347979f988dd36593eaa8 WatchSource:0}: Error finding container f5a558d2fc8fab6830e9976cab08ada076232bfead1347979f988dd36593eaa8: Status 404 returned error can't find the container with id f5a558d2fc8fab6830e9976cab08ada076232bfead1347979f988dd36593eaa8
Oct 30 13:11:36.692663 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:36.692624    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq" event={"ID":"e35f3021-173f-475c-8285-deb7716c1322","Type":"ContainerStarted","Data":"de2449d3e3c25b3161cbb4423d7e0cacd76dd5d0e73bd409dd592a10d147a3fa"}
Oct 30 13:11:36.693139 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:36.692691    4522 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq" event={"ID":"e35f3021-173f-475c-8285-deb7716c1322","Type":"ContainerStarted","Data":"f5a558d2fc8fab6830e9976cab08ada076232bfead1347979f988dd36593eaa8"}
Oct 30 13:11:36.693139 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:36.692823    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:36.733884 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:36.733837    4522 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq" podStartSLOduration=1.733823498 podStartE2EDuration="1.733823498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:36.733118609 +0000 UTC m=+2140.548445096" watchObservedRunningTime="2026-10-30 13:11:36.733823498 +0000 UTC m=+2140.549149986"
Oct 30 13:11:37.594825 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:37.594795    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fxb8p_06541c95-2fc8-432d-b39f-32434c96ea43/node-exporter/0.log"
Oct 30 13:11:37.719438 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:37.719402    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fxb8p_06541c95-2fc8-432d-b39f-32434c96ea43/kube-rbac-proxy/0.log"
Oct 30 13:11:37.842709 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:37.842676    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fxb8p_06541c95-2fc8-432d-b39f-32434c96ea43/init-textfile/0.log"
Oct 30 13:11:39.782038 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:39.782004    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-j6txf_de74b154-bc96-4ff3-8e9c-1026b658b174/prometheus-operator/0.log"
Oct 30 13:11:39.899771 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:39.899743    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-j6txf_de74b154-bc96-4ff3-8e9c-1026b658b174/kube-rbac-proxy/0.log"
Oct 30 13:11:40.023527 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:40.023485    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-hjc4m_d0cd6d0a-9b96-449a-846f-c57eb11d02e2/prometheus-operator-admission-webhook/0.log"
Oct 30 13:11:40.968272 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:40.968239    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-55f5889bd4-fv926_0ef93d47-69d0-4e66-9528-009fb281016d/manager/0.log"
Oct 30 13:11:41.193541 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:41.193510    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-55f5889bd4-fv926_0ef93d47-69d0-4e66-9528-009fb281016d/manager/0.log"
Oct 30 13:11:42.712902 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:42.712716    4522 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5tjtq/perf-node-gather-daemonset-cxppq"
Oct 30 13:11:45.255503 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:45.255471    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7657c947ff-t7nzd_bb204ee7-2696-4856-be85-1efba2443439/console/0.log"
Oct 30 13:11:45.400053 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:45.400019    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-f67bl_e93f8976-4436-4f2f-afd7-adc79bd9e975/download-server/0.log"
Oct 30 13:11:46.841699 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:46.841655    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-55f5889bd4-fv926_0ef93d47-69d0-4e66-9528-009fb281016d/manager/0.log"
Oct 30 13:11:47.134094 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:47.134019    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-55f5889bd4-fv926_0ef93d47-69d0-4e66-9528-009fb281016d/manager/0.log"
Oct 30 13:11:49.844374 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:49.844315    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x5ck2_57060355-8a18-4a06-93bb-af1de546c2b7/dns/0.log"
Oct 30 13:11:50.015095 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:50.015066    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x5ck2_57060355-8a18-4a06-93bb-af1de546c2b7/kube-rbac-proxy/0.log"
Oct 30 13:11:50.424339 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:50.424302    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5n22h_4b382eac-dd89-486d-b80b-1f5a402f23d4/dns-node-resolver/0.log"
Oct 30 13:11:50.732872 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:50.732793    4522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j8d4p_8e69cda5-58f4-409b-8575-d0d8f3c09ba6/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:51.140965 ip-10-0-129-133 kubenswrapper[4522]: E1030 13:11:51.140926    4522 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p5vwm" containerName="extractor"
Oct 30 13:11:52.158534 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:52.158480    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-674fd5846d-bn6zd_d21cf5f3-f982-4ac3-9cda-3074f3309c6e/registry/0.log"
Oct 30 13:11:52.282254 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:52.282142    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2s6fb_e14fcd05-8067-4312-b9bb-90c4c9a15b23/node-ca/0.log"
Oct 30 13:11:53.137873 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:53.137830    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kfdb7_c2c7a319-32ec-4793-8d0a-2a0b60fb8d53/discovery/0.log"
Oct 30 13:11:54.257192 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:54.257156    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-dc45784b6-l4s7t_01215889-00f7-4cfd-97cd-dee00355f9b9/kube-auth-proxy/0.log"
Oct 30 13:11:54.460707 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:54.460673    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-9twbb_f352699b-bc5c-4be3-9f32-ebdec5169271/istio-proxy/0.log"
Oct 30 13:11:54.655786 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:54.655753    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-pdt6z_f7be2765-5a1f-42db-94fd-91c0349e403d/payload-pre-processing/0.log"
Oct 30 13:11:54.725401 ip-10-0-129-133 kubenswrapper[4522]: I1030 13:11:54.725343    4522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hv6zx_4098fe5a-e7d3-488c-a1ed-72d447999e15/payload-processing/0.log"